[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250626 01:51:24.650831 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:42989,127.0.103.125:37315,127.0.103.124:34435
I20250626 01:51:24.653039 413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250626 01:51:24.654386 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:24.666313 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.666397 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.666352 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:24.667640 413 server_base.cc:1048] running on GCE node
I20250626 01:51:24.668664 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:24.668840 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:24.669435 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902684668899 us; error 0 us; skew 500 ppm
I20250626 01:51:24.669799 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:24.673115 413 webserver.cc:469] Webserver started at http://127.0.103.126:39713/ using document root <none> and password file <none>
I20250626 01:51:24.673941 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:24.674054 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:24.674456 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:24.678217 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "437a9534e69744cfa9b70794f991441f"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.685474 413 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250626 01:51:24.689487 426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:24.691390 413 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250626 01:51:24.691608 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "437a9534e69744cfa9b70794f991441f"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.691835 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:24.721939 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:24.722952 413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250626 01:51:24.723278 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:24.745967 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:42989
I20250626 01:51:24.745986 477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:42989 every 8 connection(s)
I20250626 01:51:24.749238 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:24.752050 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.752465 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.753474 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:24.754091 478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:24.754222 413 server_base.cc:1048] running on GCE node
I20250626 01:51:24.754539 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:24.754616 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:24.754693 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902684754653 us; error 0 us; skew 500 ppm
I20250626 01:51:24.754853 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:24.756347 413 webserver.cc:469] Webserver started at http://127.0.103.125:42077/ using document root <none> and password file <none>
I20250626 01:51:24.756590 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:24.756711 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:24.756852 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:24.757537 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root/instance:
uuid: "ace6c8b1793e437dad23e81bd9fb514a"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.761315 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250626 01:51:24.763492 488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:24.764278 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250626 01:51:24.764433 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root
uuid: "ace6c8b1793e437dad23e81bd9fb514a"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.764616 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:24.768901 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.787217 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:24.788074 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:24.791366 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
W20250626 01:51:24.792927 430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.125:37315: connect: Connection refused (error 111)
W20250626 01:51:24.795555 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.125:37315: Network error: Client connection negotiation failed: client connection to 127.0.103.125:37315: connect: Connection refused (error 111)
I20250626 01:51:24.808691 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:37315
I20250626 01:51:24.808722 542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.125:37315 every 8 connection(s)
I20250626 01:51:24.810362 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:51:24.810586 543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250626 01:51:24.814222 546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.814337 545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.815255 548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:24.815687 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.815834 413 server_base.cc:1048] running on GCE node
I20250626 01:51:24.816262 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:24.816362 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:24.816457 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902684816412 us; error 0 us; skew 500 ppm
I20250626 01:51:24.816658 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:24.818380 413 webserver.cc:469] Webserver started at http://127.0.103.124:39357/ using document root <none> and password file <none>
I20250626 01:51:24.818646 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:24.818770 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:24.818928 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:24.819617 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root/instance:
uuid: "15aadc52a49d44e9a3a81dcaa3d0f441"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.822993 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250626 01:51:24.823611 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.825944 554 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:24.826516 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250626 01:51:24.826678 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root
uuid: "15aadc52a49d44e9a3a81dcaa3d0f441"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.826838 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:24.833593 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
W20250626 01:51:24.836217 543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:34435: Network error: Client connection negotiation failed: client connection to 127.0.103.124:34435: connect: Connection refused (error 111)
I20250626 01:51:24.845022 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } attempt: 1
I20250626 01:51:24.852159 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.853129 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:24.853910 413 kserver.cc:163] Server-wide thread pool size limit: 3276
W20250626 01:51:24.854197 478 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:34435: Network error: Client connection negotiation failed: client connection to 127.0.103.124:34435: connect: Connection refused (error 111)
I20250626 01:51:24.872668 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:34435
I20250626 01:51:24.872709 607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.124:34435 every 8 connection(s)
I20250626 01:51:24.875125 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:51:24.875556 608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:24.878094 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.884956 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.888455 543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } attempt: 1
I20250626 01:51:24.891597 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:24.897428 478 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } attempt: 1
I20250626 01:51:24.904394 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: Bootstrap starting.
I20250626 01:51:24.904438 543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: Bootstrap starting.
I20250626 01:51:24.907956 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: Bootstrap starting.
I20250626 01:51:24.910162 608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:24.910334 543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:24.910686 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:24.912038 543 log.cc:826] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: Log is configured to *not* fsync() on all Append() calls
I20250626 01:51:24.915573 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: No bootstrap required, opened a new log
I20250626 01:51:24.915571 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: No bootstrap required, opened a new log
I20250626 01:51:24.915585 543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: No bootstrap required, opened a new log
I20250626 01:51:24.922277 608 raft_consensus.cc:357] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:24.922277 478 raft_consensus.cc:357] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:24.922278 543 raft_consensus.cc:357] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:24.923313 543 raft_consensus.cc:383] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:24.923293 478 raft_consensus.cc:383] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:24.923323 608 raft_consensus.cc:383] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:24.923535 478 raft_consensus.cc:738] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437a9534e69744cfa9b70794f991441f, State: Initialized, Role: FOLLOWER
I20250626 01:51:24.923539 608 raft_consensus.cc:738] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aadc52a49d44e9a3a81dcaa3d0f441, State: Initialized, Role: FOLLOWER
I20250626 01:51:24.923543 543 raft_consensus.cc:738] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ace6c8b1793e437dad23e81bd9fb514a, State: Initialized, Role: FOLLOWER
I20250626 01:51:24.924610 608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:24.924639 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:24.924611 543 consensus_queue.cc:260] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:24.925904 617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:24.926486 617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:24.927292 616 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:24.927778 616 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:24.927834 543 sys_catalog.cc:564] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:24.927829 478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:24.929515 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:24.929808 608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:24.930009 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:24.941816 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250626 01:51:24.942816 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250626 01:51:24.945894 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:24.949404 651 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 01:51:24.949554 651 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 01:51:24.949358 649 catalog_manager.cc:1560] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 01:51:24.949697 649 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 01:51:24.952697 648 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 01:51:24.952848 648 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 01:51:24.954692 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.956163 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:24.962677 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:24.963249 413 server_base.cc:1048] running on GCE node
I20250626 01:51:24.963984 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:24.964135 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:24.964213 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902684964189 us; error 0 us; skew 500 ppm
I20250626 01:51:24.964433 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:24.966943 413 webserver.cc:469] Webserver started at http://127.0.103.65:33999/ using document root <none> and password file <none>
I20250626 01:51:24.967341 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:24.967453 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:24.967633 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:24.968658 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "5027fc30acfc45ea96cda93aeaf0a363"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.972954 413 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250626 01:51:24.975718 661 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250626 01:51:24.975678 432 tablet.cc:2378] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:51:24.976330 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250626 01:51:24.976527 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "5027fc30acfc45ea96cda93aeaf0a363"
format_stamp: "Formatted at 2025-06-26 01:51:24 on dist-test-slave-psm5"
I20250626 01:51:24.976738 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:25.004844 616 raft_consensus.cc:491] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:25.005241 616 raft_consensus.cc:513] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:25.007653 616 leader_election.cc:290] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 437a9534e69744cfa9b70794f991441f (127.0.103.126:42989), 15aadc52a49d44e9a3a81dcaa3d0f441 (127.0.103.124:34435)
I20250626 01:51:25.008512 583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ace6c8b1793e437dad23e81bd9fb514a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" is_pre_election: true
I20250626 01:51:25.008657 453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ace6c8b1793e437dad23e81bd9fb514a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437a9534e69744cfa9b70794f991441f" is_pre_election: true
I20250626 01:51:25.009115 453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace6c8b1793e437dad23e81bd9fb514a in term 0.
I20250626 01:51:25.009119 583 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ace6c8b1793e437dad23e81bd9fb514a in term 0.
I20250626 01:51:25.009883 491 leader_election.cc:304] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437a9534e69744cfa9b70794f991441f, ace6c8b1793e437dad23e81bd9fb514a; no voters:
I20250626 01:51:25.010267 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:25.010416 616 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:25.010578 616 raft_consensus.cc:491] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:25.010697 616 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:25.011191 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:25.013159 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:25.015400 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:25.015296 616 raft_consensus.cc:513] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:25.015540 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.015671 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:25.015765 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.016779 453 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ace6c8b1793e437dad23e81bd9fb514a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437a9534e69744cfa9b70794f991441f"
I20250626 01:51:25.016930 583 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ace6c8b1793e437dad23e81bd9fb514a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441"
I20250626 01:51:25.017050 453 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:25.017184 583 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:25.021256 453 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace6c8b1793e437dad23e81bd9fb514a in term 1.
I20250626 01:51:25.022073 491 leader_election.cc:304] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437a9534e69744cfa9b70794f991441f, ace6c8b1793e437dad23e81bd9fb514a; no voters:
I20250626 01:51:25.022110 583 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ace6c8b1793e437dad23e81bd9fb514a in term 1.
I20250626 01:51:25.022902 616 leader_election.cc:290] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [CANDIDATE]: Term 1 election: Requested vote from peers 437a9534e69744cfa9b70794f991441f (127.0.103.126:42989), 15aadc52a49d44e9a3a81dcaa3d0f441 (127.0.103.124:34435)
I20250626 01:51:25.023267 616 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:25.023469 616 raft_consensus.cc:695] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 1 LEADER]: Becoming Leader. State: Replica: ace6c8b1793e437dad23e81bd9fb514a, State: Running, Role: LEADER
I20250626 01:51:25.024432 616 consensus_queue.cc:237] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } }
I20250626 01:51:25.030768 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace6c8b1793e437dad23e81bd9fb514a. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.031280 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:25.038730 671 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:51:25.042714 671 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:51:25.058019 583 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 1 FOLLOWER]: Refusing update from remote peer ace6c8b1793e437dad23e81bd9fb514a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:25.059412 669 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:25.060675 453 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 1 FOLLOWER]: Refusing update from remote peer ace6c8b1793e437dad23e81bd9fb514a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:25.063118 616 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [LEADER]: Connected to new peer: Peer: permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:25.067063 617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace6c8b1793e437dad23e81bd9fb514a. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.067385 617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:25.070536 716 mvcc.cc:204] Tried to move back new op lower bound from 7171697397979049984 to 7171697397868572672. Current Snapshot: MvccSnapshot[applied={T|T < 7171697397979049984}]
I20250626 01:51:25.088480 617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.088862 617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:25.092187 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ace6c8b1793e437dad23e81bd9fb514a. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.092579 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:25.106076 618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.106407 618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:25.113665 671 catalog_manager.cc:1349] Generated new cluster ID: e10903d0810f42eebf776f14874a1405
I20250626 01:51:25.113837 671 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:51:25.123401 616 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.123941 616 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:25.123973 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ace6c8b1793e437dad23e81bd9fb514a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "437a9534e69744cfa9b70794f991441f" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 42989 } } peers { permanent_uuid: "ace6c8b1793e437dad23e81bd9fb514a" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 37315 } } peers { permanent_uuid: "15aadc52a49d44e9a3a81dcaa3d0f441" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 34435 } } }
I20250626 01:51:25.125061 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:25.137806 671 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:51:25.139629 671 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:51:25.144160 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:36811
I20250626 01:51:25.144249 735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:36811 every 8 connection(s)
I20250626 01:51:25.172612 736 heartbeater.cc:344] Connected to a master server at 127.0.103.124:34435
I20250626 01:51:25.172964 736 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.174901 736 heartbeater.cc:507] Master 127.0.103.124:34435 requested a full tablet report, sending...
I20250626 01:51:25.178115 573 ts_manager.cc:194] Registered new tserver with Master: 5027fc30acfc45ea96cda93aeaf0a363 (127.0.103.65:36811)
I20250626 01:51:25.178705 671 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: Generated new TSK 0
I20250626 01:51:25.180347 671 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:51:25.189777 737 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42989
I20250626 01:51:25.189962 737 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.190207 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:51:25.190574 737 heartbeater.cc:507] Master 127.0.103.126:42989 requested a full tablet report, sending...
I20250626 01:51:25.192391 443 ts_manager.cc:194] Registered new tserver with Master: 5027fc30acfc45ea96cda93aeaf0a363 (127.0.103.65:36811)
W20250626 01:51:25.196836 743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:25.197938 739 heartbeater.cc:344] Connected to a master server at 127.0.103.125:37315
I20250626 01:51:25.198956 739 heartbeater.cc:461] Registering TS with master...
W20250626 01:51:25.199060 742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:25.200223 739 heartbeater.cc:507] Master 127.0.103.125:37315 requested a full tablet report, sending...
I20250626 01:51:25.201856 508 ts_manager.cc:194] Registered new tserver with Master: 5027fc30acfc45ea96cda93aeaf0a363 (127.0.103.65:36811)
W20250626 01:51:25.202831 745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:25.203671 413 server_base.cc:1048] running on GCE node
I20250626 01:51:25.204093 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:25.204186 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:25.204277 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902685204230 us; error 0 us; skew 500 ppm
I20250626 01:51:25.204464 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:25.204717 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60556
I20250626 01:51:25.208165 413 webserver.cc:469] Webserver started at http://127.0.103.66:36863/ using document root <none> and password file <none>
I20250626 01:51:25.208456 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:25.208572 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:25.208712 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:25.209460 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "d416f3307dbc42b49e4d2b936b473f61"
format_stamp: "Formatted at 2025-06-26 01:51:25 on dist-test-slave-psm5"
I20250626 01:51:25.213193 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20250626 01:51:25.215292 750 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.215776 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:51:25.215936 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "d416f3307dbc42b49e4d2b936b473f61"
format_stamp: "Formatted at 2025-06-26 01:51:25 on dist-test-slave-psm5"
I20250626 01:51:25.216115 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:25.234290 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:25.235016 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:25.235797 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:25.236961 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:25.237051 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.237191 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:25.237257 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.264547 812 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:46691 every 8 connection(s)
I20250626 01:51:25.264547 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:46691
I20250626 01:51:25.281972 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:25.292798 820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:25.303793 815 heartbeater.cc:344] Connected to a master server at 127.0.103.125:37315
I20250626 01:51:25.304069 815 heartbeater.cc:461] Registering TS with master...
W20250626 01:51:25.304075 821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:25.304744 815 heartbeater.cc:507] Master 127.0.103.125:37315 requested a full tablet report, sending...
I20250626 01:51:25.306043 508 ts_manager.cc:194] Registered new tserver with Master: d416f3307dbc42b49e4d2b936b473f61 (127.0.103.66:46691)
I20250626 01:51:25.307227 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:60570
I20250626 01:51:25.308359 813 heartbeater.cc:344] Connected to a master server at 127.0.103.124:34435
I20250626 01:51:25.308534 813 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.309795 813 heartbeater.cc:507] Master 127.0.103.124:34435 requested a full tablet report, sending...
I20250626 01:51:25.310168 814 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42989
I20250626 01:51:25.310331 814 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.310747 814 heartbeater.cc:507] Master 127.0.103.126:42989 requested a full tablet report, sending...
I20250626 01:51:25.311110 573 ts_manager.cc:194] Registered new tserver with Master: d416f3307dbc42b49e4d2b936b473f61 (127.0.103.66:46691)
I20250626 01:51:25.311831 443 ts_manager.cc:194] Registered new tserver with Master: d416f3307dbc42b49e4d2b936b473f61 (127.0.103.66:46691)
I20250626 01:51:25.312175 413 server_base.cc:1048] running on GCE node
W20250626 01:51:25.312199 823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:25.312778 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:25.312863 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:25.312930 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902685312905 us; error 0 us; skew 500 ppm
I20250626 01:51:25.313115 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:25.314787 413 webserver.cc:469] Webserver started at http://127.0.103.67:40687/ using document root <none> and password file <none>
I20250626 01:51:25.315050 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:25.315135 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:25.315308 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:25.316099 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "35a2a9fc14e84c879633b66f7f97b8a0"
format_stamp: "Formatted at 2025-06-26 01:51:25 on dist-test-slave-psm5"
I20250626 01:51:25.319118 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250626 01:51:25.321137 828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.321606 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:51:25.321753 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "35a2a9fc14e84c879633b66f7f97b8a0"
format_stamp: "Formatted at 2025-06-26 01:51:25 on dist-test-slave-psm5"
I20250626 01:51:25.321902 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:25.352056 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:25.352957 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:25.353850 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:25.355137 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:25.355255 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.355371 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:25.355460 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:25.379844 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:34411
I20250626 01:51:25.379855 890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:34411 every 8 connection(s)
I20250626 01:51:25.401981 891 heartbeater.cc:344] Connected to a master server at 127.0.103.124:34435
I20250626 01:51:25.402699 891 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.403379 891 heartbeater.cc:507] Master 127.0.103.124:34435 requested a full tablet report, sending...
I20250626 01:51:25.405283 573 ts_manager.cc:194] Registered new tserver with Master: 35a2a9fc14e84c879633b66f7f97b8a0 (127.0.103.67:34411)
I20250626 01:51:25.406239 892 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42989
I20250626 01:51:25.406428 892 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.406949 892 heartbeater.cc:507] Master 127.0.103.126:42989 requested a full tablet report, sending...
I20250626 01:51:25.408447 443 ts_manager.cc:194] Registered new tserver with Master: 35a2a9fc14e84c879633b66f7f97b8a0 (127.0.103.67:34411)
I20250626 01:51:25.413473 893 heartbeater.cc:344] Connected to a master server at 127.0.103.125:37315
I20250626 01:51:25.413753 893 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:25.414373 893 heartbeater.cc:507] Master 127.0.103.125:37315 requested a full tablet report, sending...
I20250626 01:51:25.415488 508 ts_manager.cc:194] Registered new tserver with Master: 35a2a9fc14e84c879633b66f7f97b8a0 (127.0.103.67:34411)
I20250626 01:51:25.416208 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011667806s
I20250626 01:51:25.416460 508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59302
I20250626 01:51:25.951565 651 catalog_manager.cc:1261] Loaded cluster ID: e10903d0810f42eebf776f14874a1405
I20250626 01:51:25.951680 651 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: loading cluster ID for follower catalog manager: success
I20250626 01:51:25.954348 648 catalog_manager.cc:1261] Loaded cluster ID: e10903d0810f42eebf776f14874a1405
I20250626 01:51:25.954483 648 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: loading cluster ID for follower catalog manager: success
I20250626 01:51:25.954458 651 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: acquiring CA information for follower catalog manager: success
I20250626 01:51:25.956143 651 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 01:51:25.956925 648 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: acquiring CA information for follower catalog manager: success
I20250626 01:51:25.957894 648 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 01:51:26.207517 739 heartbeater.cc:499] Master 127.0.103.125:37315 was elected leader, sending a full tablet report...
I20250626 01:51:26.309598 815 heartbeater.cc:499] Master 127.0.103.125:37315 was elected leader, sending a full tablet report...
I20250626 01:51:26.417958 893 heartbeater.cc:499] Master 127.0.103.125:37315 was elected leader, sending a full tablet report...
I20250626 01:51:26.941483 646 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:26.965489 413 test_util.cc:276] Using random seed: -860791085
I20250626 01:51:26.996273 508 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59316:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 01:51:27.042455 692 tablet_service.cc:1468] Processing CreateTablet for tablet e08e963f7a8841109d04bba82d28f6a5 (DEFAULT_TABLE table=test-workload [id=8ccc550c17ec45af85fd21d67215934a]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:51:27.042385 856 tablet_service.cc:1468] Processing CreateTablet for tablet f12c6e0426e54f44a61346ca93946cc2 (DEFAULT_TABLE table=test-workload [id=8ccc550c17ec45af85fd21d67215934a]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:51:27.042428 693 tablet_service.cc:1468] Processing CreateTablet for tablet 223c70ab92354bf2ac695db88a909492 (DEFAULT_TABLE table=test-workload [id=8ccc550c17ec45af85fd21d67215934a]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:51:27.043648 856 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f12c6e0426e54f44a61346ca93946cc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:27.043614 692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08e963f7a8841109d04bba82d28f6a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:27.049450 778 tablet_service.cc:1468] Processing CreateTablet for tablet b1cbff5f6df946b586b912bfd44f9485 (DEFAULT_TABLE table=test-workload [id=8ccc550c17ec45af85fd21d67215934a]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:51:27.050166 778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cbff5f6df946b586b912bfd44f9485. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:27.051535 693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 223c70ab92354bf2ac695db88a909492. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:27.051662 926 tablet_bootstrap.cc:492] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0: Bootstrap starting.
I20250626 01:51:27.061395 928 tablet_bootstrap.cc:492] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61: Bootstrap starting.
I20250626 01:51:27.061523 927 tablet_bootstrap.cc:492] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363: Bootstrap starting.
I20250626 01:51:27.062453 926 tablet_bootstrap.cc:654] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:27.063361 928 tablet_bootstrap.cc:654] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:27.065516 926 tablet_bootstrap.cc:492] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0: No bootstrap required, opened a new log
I20250626 01:51:27.065843 926 ts_tablet_manager.cc:1397] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.005s
I20250626 01:51:27.066180 927 tablet_bootstrap.cc:654] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:27.066277 928 tablet_bootstrap.cc:492] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61: No bootstrap required, opened a new log
I20250626 01:51:27.066428 928 ts_tablet_manager.cc:1397] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250626 01:51:27.067037 926 raft_consensus.cc:357] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a2a9fc14e84c879633b66f7f97b8a0" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34411 } }
I20250626 01:51:27.067358 928 raft_consensus.cc:357] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416f3307dbc42b49e4d2b936b473f61" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46691 } }
I20250626 01:51:27.067363 926 raft_consensus.cc:383] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:27.067570 928 raft_consensus.cc:383] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:27.067602 926 raft_consensus.cc:738] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a2a9fc14e84c879633b66f7f97b8a0, State: Initialized, Role: FOLLOWER
I20250626 01:51:27.067684 928 raft_consensus.cc:738] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d416f3307dbc42b49e4d2b936b473f61, State: Initialized, Role: FOLLOWER
I20250626 01:51:27.068012 928 consensus_queue.cc:260] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416f3307dbc42b49e4d2b936b473f61" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46691 } }
I20250626 01:51:27.068058 926 consensus_queue.cc:260] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a2a9fc14e84c879633b66f7f97b8a0" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34411 } }
I20250626 01:51:27.068599 926 raft_consensus.cc:397] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:27.068719 926 raft_consensus.cc:491] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:27.068854 926 raft_consensus.cc:3058] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:27.069211 927 tablet_bootstrap.cc:492] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363: No bootstrap required, opened a new log
I20250626 01:51:27.069384 927 ts_tablet_manager.cc:1397] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20250626 01:51:27.070175 927 raft_consensus.cc:357] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.070315 927 raft_consensus.cc:383] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:27.070384 927 raft_consensus.cc:738] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5027fc30acfc45ea96cda93aeaf0a363, State: Initialized, Role: FOLLOWER
I20250626 01:51:27.070600 927 consensus_queue.cc:260] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.070734 927 raft_consensus.cc:397] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:27.070793 927 raft_consensus.cc:491] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:27.070869 927 raft_consensus.cc:3058] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:27.071712 926 raft_consensus.cc:513] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a2a9fc14e84c879633b66f7f97b8a0" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34411 } }
I20250626 01:51:27.072000 926 leader_election.cc:304] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35a2a9fc14e84c879633b66f7f97b8a0; no voters:
I20250626 01:51:27.072420 928 raft_consensus.cc:397] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:27.072506 928 raft_consensus.cc:491] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:27.072603 928 raft_consensus.cc:3058] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:27.072798 932 raft_consensus.cc:2802] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:27.073383 932 raft_consensus.cc:695] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 1 LEADER]: Becoming Leader. State: Replica: 35a2a9fc14e84c879633b66f7f97b8a0, State: Running, Role: LEADER
I20250626 01:51:27.073745 932 consensus_queue.cc:237] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a2a9fc14e84c879633b66f7f97b8a0" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34411 } }
I20250626 01:51:27.074579 926 leader_election.cc:290] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:27.075250 926 ts_tablet_manager.cc:1428] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250626 01:51:27.075302 927 raft_consensus.cc:513] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.075544 928 raft_consensus.cc:513] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416f3307dbc42b49e4d2b936b473f61" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46691 } }
I20250626 01:51:27.075822 927 leader_election.cc:304] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5027fc30acfc45ea96cda93aeaf0a363; no voters:
I20250626 01:51:27.076620 927 leader_election.cc:290] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:27.076803 935 raft_consensus.cc:2802] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:27.075927 928 leader_election.cc:304] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d416f3307dbc42b49e4d2b936b473f61; no voters:
I20250626 01:51:27.080633 927 ts_tablet_manager.cc:1428] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363: Time spent starting tablet: real 0.011s user 0.008s sys 0.001s
I20250626 01:51:27.081123 927 tablet_bootstrap.cc:492] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363: Bootstrap starting.
I20250626 01:51:27.081056 508 catalog_manager.cc:5582] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35a2a9fc14e84c879633b66f7f97b8a0 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "35a2a9fc14e84c879633b66f7f97b8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35a2a9fc14e84c879633b66f7f97b8a0" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 34411 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:27.082300 935 raft_consensus.cc:695] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 LEADER]: Becoming Leader. State: Replica: 5027fc30acfc45ea96cda93aeaf0a363, State: Running, Role: LEADER
I20250626 01:51:27.085827 938 raft_consensus.cc:2802] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:27.085711 935 consensus_queue.cc:237] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.085508 928 leader_election.cc:290] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:27.095528 927 tablet_bootstrap.cc:654] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:27.089545 938 raft_consensus.cc:695] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 1 LEADER]: Becoming Leader. State: Replica: d416f3307dbc42b49e4d2b936b473f61, State: Running, Role: LEADER
I20250626 01:51:27.111110 938 consensus_queue.cc:237] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416f3307dbc42b49e4d2b936b473f61" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46691 } }
I20250626 01:51:27.120853 928 ts_tablet_manager.cc:1428] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61: Time spent starting tablet: real 0.051s user 0.007s sys 0.024s
I20250626 01:51:27.121104 508 catalog_manager.cc:5582] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5027fc30acfc45ea96cda93aeaf0a363 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "5027fc30acfc45ea96cda93aeaf0a363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:27.126605 507 catalog_manager.cc:5582] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to d416f3307dbc42b49e4d2b936b473f61 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "d416f3307dbc42b49e4d2b936b473f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d416f3307dbc42b49e4d2b936b473f61" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46691 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:27.135818 927 tablet_bootstrap.cc:492] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363: No bootstrap required, opened a new log
I20250626 01:51:27.136103 927 ts_tablet_manager.cc:1397] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.005s
I20250626 01:51:27.136884 927 raft_consensus.cc:357] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.137108 927 raft_consensus.cc:383] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:27.137195 927 raft_consensus.cc:738] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5027fc30acfc45ea96cda93aeaf0a363, State: Initialized, Role: FOLLOWER
I20250626 01:51:27.137454 927 consensus_queue.cc:260] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.137640 927 raft_consensus.cc:397] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:27.137750 927 raft_consensus.cc:491] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:27.137866 927 raft_consensus.cc:3058] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:27.141999 927 raft_consensus.cc:513] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.142236 927 leader_election.cc:304] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5027fc30acfc45ea96cda93aeaf0a363; no voters:
I20250626 01:51:27.142436 927 leader_election.cc:290] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:27.142493 935 raft_consensus.cc:2802] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:27.142735 935 raft_consensus.cc:695] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 LEADER]: Becoming Leader. State: Replica: 5027fc30acfc45ea96cda93aeaf0a363, State: Running, Role: LEADER
I20250626 01:51:27.143020 935 consensus_queue.cc:237] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } }
I20250626 01:51:27.143153 927 ts_tablet_manager.cc:1428] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20250626 01:51:27.145926 507 catalog_manager.cc:5582] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5027fc30acfc45ea96cda93aeaf0a363 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "5027fc30acfc45ea96cda93aeaf0a363" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fc30acfc45ea96cda93aeaf0a363" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36811 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:27.161547 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:51:27.172152 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:27.172652 413 tablet_replica.cc:331] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363: stopping tablet replica
I20250626 01:51:27.173115 413 raft_consensus.cc:2241] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:27.173422 413 raft_consensus.cc:2270] T 223c70ab92354bf2ac695db88a909492 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.175124 413 tablet_replica.cc:331] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363: stopping tablet replica
I20250626 01:51:27.175292 413 raft_consensus.cc:2241] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:27.175480 413 raft_consensus.cc:2270] T e08e963f7a8841109d04bba82d28f6a5 P 5027fc30acfc45ea96cda93aeaf0a363 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.192467 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:51:27.198165 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:51:27.207444 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:27.207851 413 tablet_replica.cc:331] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61: stopping tablet replica
I20250626 01:51:27.208108 413 raft_consensus.cc:2241] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:27.208317 413 raft_consensus.cc:2270] T b1cbff5f6df946b586b912bfd44f9485 P d416f3307dbc42b49e4d2b936b473f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.222779 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:51:27.225780 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:51:27.234401 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:27.234790 413 tablet_replica.cc:331] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0: stopping tablet replica
I20250626 01:51:27.235018 413 raft_consensus.cc:2241] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:27.235224 413 raft_consensus.cc:2270] T f12c6e0426e54f44a61346ca93946cc2 P 35a2a9fc14e84c879633b66f7f97b8a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.239516 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:51:27.242300 413 master.cc:561] Master@127.0.103.126:42989 shutting down...
I20250626 01:51:27.250741 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:27.251013 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.251171 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 437a9534e69744cfa9b70794f991441f: stopping tablet replica
I20250626 01:51:27.275116 413 master.cc:583] Master@127.0.103.126:42989 shutdown complete.
I20250626 01:51:27.280258 413 master.cc:561] Master@127.0.103.125:37315 shutting down...
I20250626 01:51:27.288066 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:27.288602 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.288746 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P ace6c8b1793e437dad23e81bd9fb514a: stopping tablet replica
I20250626 01:51:27.304181 413 master.cc:583] Master@127.0.103.125:37315 shutdown complete.
I20250626 01:51:27.308691 413 master.cc:561] Master@127.0.103.124:34435 shutting down...
I20250626 01:51:27.316430 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:27.316689 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:27.316810 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 15aadc52a49d44e9a3a81dcaa3d0f441: stopping tablet replica
I20250626 01:51:27.330960 413 master.cc:583] Master@127.0.103.124:34435 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (2697 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250626 01:51:27.345278 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41023
I20250626 01:51:27.345865 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:27.348306 948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:27.348439 949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:27.349727 413 server_base.cc:1048] running on GCE node
W20250626 01:51:27.349771 951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:27.350164 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:27.350263 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:27.350323 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902687350303 us; error 0 us; skew 500 ppm
I20250626 01:51:27.350498 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:27.351915 413 webserver.cc:469] Webserver started at http://127.0.103.126:33935/ using document root <none> and password file <none>
I20250626 01:51:27.352202 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:27.352289 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:27.352448 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:27.353149 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "96a70ca2c9b64ecc8846e1a1de62594f"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.356005 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250626 01:51:27.357893 956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.358295 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250626 01:51:27.358417 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "96a70ca2c9b64ecc8846e1a1de62594f"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.358543 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:27.380120 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:27.380811 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:27.397032 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41023
I20250626 01:51:27.397050 1007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41023 every 8 connection(s)
I20250626 01:51:27.398617 1008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:27.402889 1008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f: Bootstrap starting.
I20250626 01:51:27.404471 1008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:27.406292 1008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f: No bootstrap required, opened a new log
I20250626 01:51:27.406967 1008 raft_consensus.cc:357] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" member_type: VOTER }
I20250626 01:51:27.407117 1008 raft_consensus.cc:383] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:27.407179 1008 raft_consensus.cc:738] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a70ca2c9b64ecc8846e1a1de62594f, State: Initialized, Role: FOLLOWER
I20250626 01:51:27.407379 1008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" member_type: VOTER }
I20250626 01:51:27.407511 1008 raft_consensus.cc:397] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:27.407572 1008 raft_consensus.cc:491] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:27.407653 1008 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:27.409596 1008 raft_consensus.cc:513] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" member_type: VOTER }
I20250626 01:51:27.409828 1008 leader_election.cc:304] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96a70ca2c9b64ecc8846e1a1de62594f; no voters:
I20250626 01:51:27.410255 1008 leader_election.cc:290] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:27.410460 1011 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:27.410993 1011 raft_consensus.cc:695] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 1 LEADER]: Becoming Leader. State: Replica: 96a70ca2c9b64ecc8846e1a1de62594f, State: Running, Role: LEADER
I20250626 01:51:27.411226 1008 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:27.411274 1011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" member_type: VOTER }
I20250626 01:51:27.412350 1012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" member_type: VOTER } }
I20250626 01:51:27.412434 1013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a70ca2c9b64ecc8846e1a1de62594f. Latest consensus state: current_term: 1 leader_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96a70ca2c9b64ecc8846e1a1de62594f" member_type: VOTER } }
I20250626 01:51:27.412592 1012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:27.412622 1013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:27.413764 1016 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:51:27.415408 1016 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:51:27.417582 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:51:27.418793 1016 catalog_manager.cc:1349] Generated new cluster ID: cfaa6eb122f6439dab182619f37e7f77
I20250626 01:51:27.418895 1016 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:51:27.433938 1016 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:51:27.434777 1016 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:51:27.450201 1016 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f: Generated new TSK 0
I20250626 01:51:27.450495 1016 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:51:27.482905 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:27.486243 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:27.486370 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:27.487782 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:27.487852 413 server_base.cc:1048] running on GCE node
I20250626 01:51:27.488257 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:27.488345 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:27.488435 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902687488394 us; error 0 us; skew 500 ppm
I20250626 01:51:27.488636 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:27.490063 413 webserver.cc:469] Webserver started at http://127.0.103.65:38461/ using document root <none> and password file <none>
I20250626 01:51:27.490307 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:27.490408 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:27.490557 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:27.491282 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.494076 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250626 01:51:27.495740 1037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.496198 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250626 01:51:27.496323 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.496449 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:27.518096 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:27.518729 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:27.519309 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:27.520291 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:27.520370 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.520483 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:27.520545 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.538300 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:33203
I20250626 01:51:27.538350 1099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:33203 every 8 connection(s)
I20250626 01:51:27.540236 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:27.544425 1104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:27.544431 1105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:27.545022 1107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:27.546543 413 server_base.cc:1048] running on GCE node
I20250626 01:51:27.546893 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:27.547013 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:27.547101 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902687547064 us; error 0 us; skew 500 ppm
I20250626 01:51:27.547326 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:27.548884 413 webserver.cc:469] Webserver started at http://127.0.103.66:39701/ using document root <none> and password file <none>
I20250626 01:51:27.549206 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:27.549305 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:27.549515 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:27.549583 1100 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41023
I20250626 01:51:27.549762 1100 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:27.550165 1100 heartbeater.cc:507] Master 127.0.103.126:41023 requested a full tablet report, sending...
I20250626 01:51:27.550299 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "37a6f624375545bd81ce4d8d11635112"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.551190 973 ts_manager.cc:194] Registered new tserver with Master: 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203)
I20250626 01:51:27.552384 973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39566
I20250626 01:51:27.553324 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:51:27.555222 1112 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.555581 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:51:27.555696 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "37a6f624375545bd81ce4d8d11635112"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.555812 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:27.571280 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:27.571981 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:27.572588 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:27.573417 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:27.573511 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.573617 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:27.573698 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.591219 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:33771
I20250626 01:51:27.591265 1174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:33771 every 8 connection(s)
I20250626 01:51:27.592991 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:27.597360 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:27.598042 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:27.600162 413 server_base.cc:1048] running on GCE node
W20250626 01:51:27.600772 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:27.601265 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:27.601404 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:27.601635 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902687601452 us; error 0 us; skew 500 ppm
I20250626 01:51:27.601814 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:27.602684 1175 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41023
I20250626 01:51:27.602905 1175 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:27.603401 1175 heartbeater.cc:507] Master 127.0.103.126:41023 requested a full tablet report, sending...
I20250626 01:51:27.604162 413 webserver.cc:469] Webserver started at http://127.0.103.67:42499/ using document root <none> and password file <none>
I20250626 01:51:27.604413 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:27.604527 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:27.604671 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:27.604996 973 ts_manager.cc:194] Registered new tserver with Master: 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771)
I20250626 01:51:27.605379 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "a050999cf24a4c7dbe77f664a561b6c3"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.605883 973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39582
I20250626 01:51:27.608007 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250626 01:51:27.609727 1186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.610142 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:51:27.610240 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "a050999cf24a4c7dbe77f664a561b6c3"
format_stamp: "Formatted at 2025-06-26 01:51:27 on dist-test-slave-psm5"
I20250626 01:51:27.610364 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:27.622323 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:27.622956 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:27.624078 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:27.624969 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:27.625037 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.625131 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:27.625175 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:27.639703 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:38159
I20250626 01:51:27.639734 1248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:38159 every 8 connection(s)
I20250626 01:51:27.646726 1249 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41023
I20250626 01:51:27.646903 1249 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:27.647312 1249 heartbeater.cc:507] Master 127.0.103.126:41023 requested a full tablet report, sending...
I20250626 01:51:27.648224 973 ts_manager.cc:194] Registered new tserver with Master: a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:27.648495 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007862951s
I20250626 01:51:27.649111 973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39592
I20250626 01:51:28.554039 1100 heartbeater.cc:499] Master 127.0.103.126:41023 was elected leader, sending a full tablet report...
I20250626 01:51:28.607358 1175 heartbeater.cc:499] Master 127.0.103.126:41023 was elected leader, sending a full tablet report...
I20250626 01:51:28.650405 1249 heartbeater.cc:499] Master 127.0.103.126:41023 was elected leader, sending a full tablet report...
I20250626 01:51:28.676642 413 test_util.cc:276] Using random seed: -859079924
I20250626 01:51:28.692761 973 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39596:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:51:28.693586 973 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:51:28.719054 1065 tablet_service.cc:1468] Processing CreateTablet for tablet 3039c4c5043945c29d4a559f79defe08 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250626 01:51:28.719446 1063 tablet_service.cc:1468] Processing CreateTablet for tablet 39fd63d1d7354ab09ae3c1845ba3a4a9 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250626 01:51:28.719732 1065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3039c4c5043945c29d4a559f79defe08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.719872 1061 tablet_service.cc:1468] Processing CreateTablet for tablet 6e65bf8cdda3438da969f92e5b4a64b8 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250626 01:51:28.720563 1063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fd63d1d7354ab09ae3c1845ba3a4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.720938 1061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e65bf8cdda3438da969f92e5b4a64b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.721496 1059 tablet_service.cc:1468] Processing CreateTablet for tablet 1032027a759344af84de0ca88d05566f (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250626 01:51:28.721983 1059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1032027a759344af84de0ca88d05566f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.720283 1058 tablet_service.cc:1468] Processing CreateTablet for tablet 60acdfbdcf5b46fd885cbbe77992b763 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250626 01:51:28.727692 1058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60acdfbdcf5b46fd885cbbe77992b763. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.727985 1064 tablet_service.cc:1468] Processing CreateTablet for tablet 74bdae8a7f4a436cabd708d053559461 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250626 01:51:28.728338 1062 tablet_service.cc:1468] Processing CreateTablet for tablet 4964c45e0491413c9baf98039bf2557f (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250626 01:51:28.728657 1060 tablet_service.cc:1468] Processing CreateTablet for tablet 29908d5bcce84f7499b9a269e8f6e6f7 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250626 01:51:28.730954 1064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74bdae8a7f4a436cabd708d053559461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.732367 1062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4964c45e0491413c9baf98039bf2557f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.736513 1140 tablet_service.cc:1468] Processing CreateTablet for tablet 3039c4c5043945c29d4a559f79defe08 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250626 01:51:28.737118 1140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3039c4c5043945c29d4a559f79defe08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.736716 1060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29908d5bcce84f7499b9a269e8f6e6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.740689 1269 tablet_bootstrap.cc:492] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.741307 1139 tablet_service.cc:1468] Processing CreateTablet for tablet 74bdae8a7f4a436cabd708d053559461 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250626 01:51:28.741915 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74bdae8a7f4a436cabd708d053559461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.742950 1269 tablet_bootstrap.cc:654] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.744441 1138 tablet_service.cc:1468] Processing CreateTablet for tablet 39fd63d1d7354ab09ae3c1845ba3a4a9 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250626 01:51:28.745033 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fd63d1d7354ab09ae3c1845ba3a4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.749346 1137 tablet_service.cc:1468] Processing CreateTablet for tablet 4964c45e0491413c9baf98039bf2557f (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250626 01:51:28.750361 1134 tablet_service.cc:1468] Processing CreateTablet for tablet 1032027a759344af84de0ca88d05566f (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250626 01:51:28.751222 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4964c45e0491413c9baf98039bf2557f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.751259 1135 tablet_service.cc:1468] Processing CreateTablet for tablet 29908d5bcce84f7499b9a269e8f6e6f7 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250626 01:51:28.753625 1134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1032027a759344af84de0ca88d05566f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.756011 1269 tablet_bootstrap.cc:492] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.755980 1136 tablet_service.cc:1468] Processing CreateTablet for tablet 6e65bf8cdda3438da969f92e5b4a64b8 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250626 01:51:28.756227 1269 ts_tablet_manager.cc:1397] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.008s
I20250626 01:51:28.754981 1133 tablet_service.cc:1468] Processing CreateTablet for tablet 60acdfbdcf5b46fd885cbbe77992b763 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250626 01:51:28.757578 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29908d5bcce84f7499b9a269e8f6e6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.757611 1212 tablet_service.cc:1468] Processing CreateTablet for tablet 39fd63d1d7354ab09ae3c1845ba3a4a9 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250626 01:51:28.757853 1133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60acdfbdcf5b46fd885cbbe77992b763. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.758698 1214 tablet_service.cc:1468] Processing CreateTablet for tablet 3039c4c5043945c29d4a559f79defe08 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250626 01:51:28.758978 1210 tablet_service.cc:1468] Processing CreateTablet for tablet 6e65bf8cdda3438da969f92e5b4a64b8 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250626 01:51:28.759936 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e65bf8cdda3438da969f92e5b4a64b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.759970 1211 tablet_service.cc:1468] Processing CreateTablet for tablet 4964c45e0491413c9baf98039bf2557f (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250626 01:51:28.760133 1212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39fd63d1d7354ab09ae3c1845ba3a4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.760962 1209 tablet_service.cc:1468] Processing CreateTablet for tablet 29908d5bcce84f7499b9a269e8f6e6f7 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250626 01:51:28.762084 1214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3039c4c5043945c29d4a559f79defe08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.763484 1213 tablet_service.cc:1468] Processing CreateTablet for tablet 74bdae8a7f4a436cabd708d053559461 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250626 01:51:28.763970 1211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4964c45e0491413c9baf98039bf2557f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.761895 1208 tablet_service.cc:1468] Processing CreateTablet for tablet 1032027a759344af84de0ca88d05566f (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250626 01:51:28.764842 1210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e65bf8cdda3438da969f92e5b4a64b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.764030 1207 tablet_service.cc:1468] Processing CreateTablet for tablet 60acdfbdcf5b46fd885cbbe77992b763 (DEFAULT_TABLE table=test-workload [id=cb79b7b908d048d3a307f5ec957d4eac]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250626 01:51:28.767489 1213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74bdae8a7f4a436cabd708d053559461. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.769552 1209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29908d5bcce84f7499b9a269e8f6e6f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.769840 1208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1032027a759344af84de0ca88d05566f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.770107 1207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60acdfbdcf5b46fd885cbbe77992b763. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:28.769374 1269 raft_consensus.cc:357] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.772446 1269 raft_consensus.cc:383] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.772585 1269 raft_consensus.cc:738] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.772868 1269 consensus_queue.cc:260] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.821362 1272 tablet_bootstrap.cc:492] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.823211 1272 tablet_bootstrap.cc:654] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.824942 1269 ts_tablet_manager.cc:1428] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.069s user 0.011s sys 0.022s
I20250626 01:51:28.828214 1269 tablet_bootstrap.cc:492] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.828831 1272 tablet_bootstrap.cc:492] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:28.829001 1272 ts_tablet_manager.cc:1397] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250626 01:51:28.829941 1272 raft_consensus.cc:357] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.830216 1272 raft_consensus.cc:383] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.830278 1269 tablet_bootstrap.cc:654] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.830331 1273 tablet_bootstrap.cc:492] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:28.830353 1272 raft_consensus.cc:738] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.830710 1272 consensus_queue.cc:260] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.831746 1272 ts_tablet_manager.cc:1428] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250626 01:51:28.832181 1272 tablet_bootstrap.cc:492] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.832502 1273 tablet_bootstrap.cc:654] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.835529 1272 tablet_bootstrap.cc:654] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.837867 1273 tablet_bootstrap.cc:492] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:28.838064 1273 ts_tablet_manager.cc:1397] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20250626 01:51:28.838987 1273 raft_consensus.cc:357] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.839222 1273 raft_consensus.cc:383] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.839339 1273 raft_consensus.cc:738] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.839617 1273 consensus_queue.cc:260] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.840607 1273 ts_tablet_manager.cc:1428] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250626 01:51:28.841099 1273 tablet_bootstrap.cc:492] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:28.841620 1269 tablet_bootstrap.cc:492] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.841849 1269 ts_tablet_manager.cc:1397] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250626 01:51:28.843106 1273 tablet_bootstrap.cc:654] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.842960 1269 raft_consensus.cc:357] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.843269 1269 raft_consensus.cc:383] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.843427 1269 raft_consensus.cc:738] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.843758 1269 consensus_queue.cc:260] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.844478 1269 ts_tablet_manager.cc:1428] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:51:28.844918 1269 tablet_bootstrap.cc:492] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.846980 1269 tablet_bootstrap.cc:654] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.848205 1272 tablet_bootstrap.cc:492] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:28.848423 1272 ts_tablet_manager.cc:1397] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20250626 01:51:28.849575 1272 raft_consensus.cc:357] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.849879 1272 raft_consensus.cc:383] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.849988 1272 raft_consensus.cc:738] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.850086 1269 tablet_bootstrap.cc:492] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.850258 1269 ts_tablet_manager.cc:1397] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250626 01:51:28.850360 1272 consensus_queue.cc:260] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.851030 1269 raft_consensus.cc:357] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.851218 1269 raft_consensus.cc:383] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.851356 1269 raft_consensus.cc:738] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.851640 1269 consensus_queue.cc:260] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.852203 1269 ts_tablet_manager.cc:1428] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:51:28.852259 1272 ts_tablet_manager.cc:1428] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250626 01:51:28.852495 1273 tablet_bootstrap.cc:492] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:28.852527 1269 tablet_bootstrap.cc:492] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.852556 1272 tablet_bootstrap.cc:492] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.852669 1273 ts_tablet_manager.cc:1397] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20250626 01:51:28.854593 1272 tablet_bootstrap.cc:654] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.854643 1269 tablet_bootstrap.cc:654] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.857468 1273 raft_consensus.cc:357] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.857728 1273 raft_consensus.cc:383] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.857867 1273 raft_consensus.cc:738] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.858281 1273 consensus_queue.cc:260] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.862056 1273 ts_tablet_manager.cc:1428] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250626 01:51:28.863751 1273 tablet_bootstrap.cc:492] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:28.865479 1269 tablet_bootstrap.cc:492] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.865667 1269 ts_tablet_manager.cc:1397] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.009s
I20250626 01:51:28.866362 1269 raft_consensus.cc:357] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.866546 1273 tablet_bootstrap.cc:654] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.866566 1269 raft_consensus.cc:383] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.866709 1269 raft_consensus.cc:738] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.866982 1269 consensus_queue.cc:260] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.867492 1269 ts_tablet_manager.cc:1428] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:51:28.867866 1269 tablet_bootstrap.cc:492] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.869938 1269 tablet_bootstrap.cc:654] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.870783 1273 tablet_bootstrap.cc:492] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:28.870939 1273 ts_tablet_manager.cc:1397] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250626 01:51:28.871924 1273 raft_consensus.cc:357] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.872174 1273 raft_consensus.cc:383] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.872268 1273 raft_consensus.cc:738] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.872610 1273 consensus_queue.cc:260] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.872970 1269 tablet_bootstrap.cc:492] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.873132 1269 ts_tablet_manager.cc:1397] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250626 01:51:28.873857 1273 ts_tablet_manager.cc:1428] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250626 01:51:28.874020 1269 raft_consensus.cc:357] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.874202 1269 raft_consensus.cc:383] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.874320 1269 raft_consensus.cc:738] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.874181 1273 tablet_bootstrap.cc:492] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:28.874576 1269 consensus_queue.cc:260] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.875123 1269 ts_tablet_manager.cc:1428] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:51:28.875437 1269 tablet_bootstrap.cc:492] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.877712 1279 raft_consensus.cc:491] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:28.878895 1272 tablet_bootstrap.cc:492] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:28.878001 1279 raft_consensus.cc:513] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.879369 1272 ts_tablet_manager.cc:1397] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.013s
I20250626 01:51:28.880990 1273 tablet_bootstrap.cc:654] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.881253 1279 leader_election.cc:290] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771)
I20250626 01:51:28.881151 1272 raft_consensus.cc:357] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.881389 1272 raft_consensus.cc:383] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.881515 1272 raft_consensus.cc:738] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.882166 1269 tablet_bootstrap.cc:654] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.881843 1272 consensus_queue.cc:260] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.883071 1272 ts_tablet_manager.cc:1428] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250626 01:51:28.883387 1272 tablet_bootstrap.cc:492] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.885336 1272 tablet_bootstrap.cc:654] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.886174 1271 raft_consensus.cc:491] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:28.886336 1271 raft_consensus.cc:513] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.887454 1271 leader_election.cc:290] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:28.907455 1272 tablet_bootstrap.cc:492] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:28.909155 1272 ts_tablet_manager.cc:1397] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250626 01:51:28.911522 1272 raft_consensus.cc:357] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.912535 1272 raft_consensus.cc:383] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.914052 1272 raft_consensus.cc:738] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.914628 1272 consensus_queue.cc:260] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.915910 1269 tablet_bootstrap.cc:492] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.916581 1272 ts_tablet_manager.cc:1428] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250626 01:51:28.918354 1273 tablet_bootstrap.cc:492] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:28.918404 1272 tablet_bootstrap.cc:492] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.918565 1273 ts_tablet_manager.cc:1397] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.013s
I20250626 01:51:28.919612 1273 raft_consensus.cc:357] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.919857 1273 raft_consensus.cc:383] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.919981 1273 raft_consensus.cc:738] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.920857 1269 ts_tablet_manager.cc:1397] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.006s
I20250626 01:51:28.921525 1272 tablet_bootstrap.cc:654] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.922572 1273 consensus_queue.cc:260] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.923504 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3039c4c5043945c29d4a559f79defe08" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" is_pre_election: true
I20250626 01:51:28.924539 1273 ts_tablet_manager.cc:1428] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250626 01:51:28.924579 1075 raft_consensus.cc:2466] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a050999cf24a4c7dbe77f664a561b6c3 in term 0.
I20250626 01:51:28.925848 1189 leader_election.cc:304] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:28.925809 1273 tablet_bootstrap.cc:492] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:28.926307 1279 raft_consensus.cc:2802] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:28.926761 1279 raft_consensus.cc:491] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:28.927150 1279 raft_consensus.cc:3058] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:28.928790 1269 raft_consensus.cc:357] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.931608 1269 raft_consensus.cc:383] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.930438 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3039c4c5043945c29d4a559f79defe08" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:28.931744 1269 raft_consensus.cc:738] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
W20250626 01:51:28.933197 1187 leader_election.cc:343] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:51:28.933744 1269 consensus_queue.cc:260] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.936914 1279 raft_consensus.cc:513] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.937849 1269 ts_tablet_manager.cc:1428] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20250626 01:51:28.938172 1279 leader_election.cc:290] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 election: Requested vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771)
I20250626 01:51:28.938336 1269 tablet_bootstrap.cc:492] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.939628 1272 tablet_bootstrap.cc:492] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:28.939814 1272 ts_tablet_manager.cc:1397] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20250626 01:51:28.939922 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3039c4c5043945c29d4a559f79defe08" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
I20250626 01:51:28.940155 1075 raft_consensus.cc:3058] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:28.940821 1269 tablet_bootstrap.cc:654] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.941625 1272 raft_consensus.cc:357] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.938387 1273 tablet_bootstrap.cc:654] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.941857 1272 raft_consensus.cc:383] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.941963 1272 raft_consensus.cc:738] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.942642 1075 raft_consensus.cc:2466] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050999cf24a4c7dbe77f664a561b6c3 in term 1.
I20250626 01:51:28.943009 1271 raft_consensus.cc:491] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:28.943171 1271 raft_consensus.cc:513] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.943352 1272 consensus_queue.cc:260] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.943822 1271 leader_election.cc:290] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:28.944468 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3039c4c5043945c29d4a559f79defe08" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
W20250626 01:51:28.945003 1187 leader_election.cc:343] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:51:28.945092 1272 ts_tablet_manager.cc:1428] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250626 01:51:28.945497 1273 tablet_bootstrap.cc:492] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:28.945636 1272 tablet_bootstrap.cc:492] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.945761 1273 ts_tablet_manager.cc:1397] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20250626 01:51:28.946736 1273 raft_consensus.cc:357] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.947033 1273 raft_consensus.cc:383] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.947192 1273 raft_consensus.cc:738] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.947743 1189 leader_election.cc:304] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters: 37a6f624375545bd81ce4d8d11635112
I20250626 01:51:28.948123 1279 raft_consensus.cc:2802] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:28.952593 1272 tablet_bootstrap.cc:654] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.953650 1273 consensus_queue.cc:260] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.957125 1279 raft_consensus.cc:695] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 LEADER]: Becoming Leader. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Running, Role: LEADER
I20250626 01:51:28.959507 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:28.959447 1279 consensus_queue.cc:237] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:28.960556 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74bdae8a7f4a436cabd708d053559461" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
W20250626 01:51:28.960973 1038 leader_election.cc:343] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:51:28.960950 1273 ts_tablet_manager.cc:1428] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.015s user 0.001s sys 0.002s
I20250626 01:51:28.961741 1273 tablet_bootstrap.cc:492] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
W20250626 01:51:28.963186 1038 leader_election.cc:343] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:51:28.963492 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:28.963735 1150 raft_consensus.cc:2466] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:28.964255 1038 leader_election.cc:304] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters: a050999cf24a4c7dbe77f664a561b6c3
I20250626 01:51:28.964951 1271 raft_consensus.cc:2802] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:28.965080 1271 raft_consensus.cc:491] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:28.965415 1271 raft_consensus.cc:3058] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:28.965534 1269 tablet_bootstrap.cc:492] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:28.965766 1269 ts_tablet_manager.cc:1397] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.028s user 0.014s sys 0.010s
I20250626 01:51:28.967039 1269 raft_consensus.cc:357] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.967352 1269 raft_consensus.cc:383] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.967489 1269 raft_consensus.cc:738] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.967825 1271 raft_consensus.cc:513] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.969491 1269 consensus_queue.cc:260] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.972658 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:28.972736 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:28.973042 1150 raft_consensus.cc:3058] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:28.968883 1149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74bdae8a7f4a436cabd708d053559461" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:28.973474 1271 leader_election.cc:290] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:28.973621 1149 raft_consensus.cc:2466] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
W20250626 01:51:28.974345 1038 leader_election.cc:343] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:51:28.976104 1038 leader_election.cc:304] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters: a050999cf24a4c7dbe77f664a561b6c3
I20250626 01:51:28.974390 971 catalog_manager.cc:5582] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "a050999cf24a4c7dbe77f664a561b6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:28.977385 1271 raft_consensus.cc:2802] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:28.978196 1271 raft_consensus.cc:491] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:28.978709 1271 raft_consensus.cc:3058] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:28.979758 1272 tablet_bootstrap.cc:492] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:28.980326 1272 ts_tablet_manager.cc:1397] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20250626 01:51:28.980630 1269 ts_tablet_manager.cc:1428] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
I20250626 01:51:28.980980 1269 tablet_bootstrap.cc:492] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: Bootstrap starting.
I20250626 01:51:28.981302 1272 raft_consensus.cc:357] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.981540 1272 raft_consensus.cc:383] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.981635 1272 raft_consensus.cc:738] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.982009 1272 consensus_queue.cc:260] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.982823 1272 ts_tablet_manager.cc:1428] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:51:28.984432 1272 tablet_bootstrap.cc:492] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:28.985236 1150 raft_consensus.cc:2466] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:28.986117 1273 tablet_bootstrap.cc:654] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.986037 1271 raft_consensus.cc:513] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.986124 1038 leader_election.cc:304] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters: a050999cf24a4c7dbe77f664a561b6c3
I20250626 01:51:28.986500 1301 raft_consensus.cc:2802] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:28.988276 1271 leader_election.cc:290] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:28.988982 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74bdae8a7f4a436cabd708d053559461" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:28.989843 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74bdae8a7f4a436cabd708d053559461" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
W20250626 01:51:28.989862 1038 leader_election.cc:343] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:51:28.990303 1150 raft_consensus.cc:3058] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:28.991425 1301 raft_consensus.cc:695] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:28.992290 1301 consensus_queue.cc:237] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.993821 1302 raft_consensus.cc:491] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:28.994575 1269 tablet_bootstrap.cc:654] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.994835 1273 tablet_bootstrap.cc:492] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:28.995054 1273 ts_tablet_manager.cc:1397] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.003s
I20250626 01:51:28.994134 1302 raft_consensus.cc:513] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.995939 1273 raft_consensus.cc:357] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.995997 1150 raft_consensus.cc:2466] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:28.996191 1273 raft_consensus.cc:383] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:28.996348 1273 raft_consensus.cc:738] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:28.996654 1273 consensus_queue.cc:260] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:28.996810 1272 tablet_bootstrap.cc:654] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:28.996923 1038 leader_election.cc:304] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters: a050999cf24a4c7dbe77f664a561b6c3
I20250626 01:51:28.997267 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39fd63d1d7354ab09ae3c1845ba3a4a9" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:28.997438 1271 raft_consensus.cc:2802] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:28.997558 1150 raft_consensus.cc:2466] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:28.998629 1038 leader_election.cc:304] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters:
I20250626 01:51:28.998889 1273 ts_tablet_manager.cc:1428] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250626 01:51:28.999310 1273 tablet_bootstrap.cc:492] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:29.002166 1273 tablet_bootstrap.cc:654] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:29.004290 1302 leader_election.cc:290] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.004623 1302 raft_consensus.cc:2802] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:29.005079 1302 raft_consensus.cc:491] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:29.005280 1302 raft_consensus.cc:3058] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.009908 1272 tablet_bootstrap.cc:492] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:29.010178 1272 ts_tablet_manager.cc:1397] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20250626 01:51:29.010368 1302 raft_consensus.cc:513] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.011515 1302 leader_election.cc:290] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.011623 1272 raft_consensus.cc:357] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:29.012307 1272 raft_consensus.cc:383] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:29.012485 1272 raft_consensus.cc:738] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:29.012794 1272 consensus_queue.cc:260] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:29.012956 1271 raft_consensus.cc:695] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.013263 1271 consensus_queue.cc:237] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.017174 1272 ts_tablet_manager.cc:1428] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250626 01:51:29.020247 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39fd63d1d7354ab09ae3c1845ba3a4a9" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:29.020380 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39fd63d1d7354ab09ae3c1845ba3a4a9" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:29.020560 1223 raft_consensus.cc:2466] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.020668 1150 raft_consensus.cc:3058] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.020949 1272 tablet_bootstrap.cc:492] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112: Bootstrap starting.
I20250626 01:51:29.024857 1269 tablet_bootstrap.cc:492] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: No bootstrap required, opened a new log
I20250626 01:51:29.025310 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "39fd63d1d7354ab09ae3c1845ba3a4a9" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.025324 1269 ts_tablet_manager.cc:1397] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.001s
I20250626 01:51:29.025877 1223 raft_consensus.cc:3058] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.030031 1150 raft_consensus.cc:2466] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.030912 1038 leader_election.cc:304] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters:
I20250626 01:51:29.031517 1269 raft_consensus.cc:357] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.031919 1271 raft_consensus.cc:2802] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:29.032238 1269 raft_consensus.cc:383] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:29.032449 1269 raft_consensus.cc:738] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Initialized, Role: FOLLOWER
I20250626 01:51:29.032477 1271 raft_consensus.cc:695] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.032925 1269 consensus_queue.cc:260] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.032928 1271 consensus_queue.cc:237] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.033573 1269 ts_tablet_manager.cc:1428] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250626 01:51:29.033972 1223 raft_consensus.cc:2466] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.035624 1272 tablet_bootstrap.cc:654] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:29.036899 1273 tablet_bootstrap.cc:492] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:29.037073 1273 ts_tablet_manager.cc:1397] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.038s user 0.012s sys 0.020s
I20250626 01:51:29.037900 1273 raft_consensus.cc:357] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.038082 1273 raft_consensus.cc:383] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:29.038187 1273 raft_consensus.cc:738] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:29.038429 1273 consensus_queue.cc:260] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.038317 971 catalog_manager.cc:5582] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.038969 1273 ts_tablet_manager.cc:1428] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:51:29.039637 1273 tablet_bootstrap.cc:492] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3: Bootstrap starting.
I20250626 01:51:29.040511 1272 tablet_bootstrap.cc:492] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112: No bootstrap required, opened a new log
I20250626 01:51:29.041262 1272 ts_tablet_manager.cc:1397] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.002s
I20250626 01:51:29.041457 1273 tablet_bootstrap.cc:654] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:29.042165 1272 raft_consensus.cc:357] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.042397 1272 raft_consensus.cc:383] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:29.042500 1272 raft_consensus.cc:738] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Initialized, Role: FOLLOWER
I20250626 01:51:29.042747 1272 consensus_queue.cc:260] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.043359 1272 ts_tablet_manager.cc:1428] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:51:29.045837 972 catalog_manager.cc:5582] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.046324 1273 tablet_bootstrap.cc:492] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3: No bootstrap required, opened a new log
I20250626 01:51:29.046623 1273 ts_tablet_manager.cc:1397] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20250626 01:51:29.046842 972 catalog_manager.cc:5582] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.047546 1273 raft_consensus.cc:357] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.047785 1273 raft_consensus.cc:383] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:29.047915 1273 raft_consensus.cc:738] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Initialized, Role: FOLLOWER
I20250626 01:51:29.048202 1273 consensus_queue.cc:260] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.048760 1273 ts_tablet_manager.cc:1428] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:51:29.061762 1301 raft_consensus.cc:491] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.062023 1301 raft_consensus.cc:513] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.062693 1301 leader_election.cc:290] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.063230 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60acdfbdcf5b46fd885cbbe77992b763" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:29.063251 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60acdfbdcf5b46fd885cbbe77992b763" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:29.063463 1223 raft_consensus.cc:2466] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.063482 1150 raft_consensus.cc:2466] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.063853 1038 leader_election.cc:304] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.064158 1301 raft_consensus.cc:2802] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:29.064240 1301 raft_consensus.cc:491] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:29.064352 1301 raft_consensus.cc:3058] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.066879 1301 raft_consensus.cc:513] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.067504 1301 leader_election.cc:290] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.068126 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60acdfbdcf5b46fd885cbbe77992b763" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.068121 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60acdfbdcf5b46fd885cbbe77992b763" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:29.068369 1223 raft_consensus.cc:3058] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.068425 1150 raft_consensus.cc:3058] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.070377 1279 raft_consensus.cc:491] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.070554 1150 raft_consensus.cc:2466] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.070606 1223 raft_consensus.cc:2466] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.070634 1279 raft_consensus.cc:513] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.071020 1038 leader_election.cc:304] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.071321 1301 raft_consensus.cc:2802] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:29.071429 1279 leader_election.cc:290] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771)
I20250626 01:51:29.071502 1301 raft_consensus.cc:695] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.071822 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" is_pre_election: true
I20250626 01:51:29.072052 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:29.072294 1301 consensus_queue.cc:237] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.072429 1150 raft_consensus.cc:2391] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a050999cf24a4c7dbe77f664a561b6c3 in current term 1: Already voted for candidate 6bff29e811ab4f89b8698bcc7553cc5a in this term.
I20250626 01:51:29.072888 1187 leader_election.cc:304] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a050999cf24a4c7dbe77f664a561b6c3; no voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a
I20250626 01:51:29.073302 1279 raft_consensus.cc:3058] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.075734 1279 raft_consensus.cc:2747] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 01:51:29.075582 971 catalog_manager.cc:5582] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.130759 1301 raft_consensus.cc:491] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.131002 1301 raft_consensus.cc:513] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.131778 1301 leader_election.cc:290] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.132424 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:29.132428 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:29.132732 1223 raft_consensus.cc:2466] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.132763 1150 raft_consensus.cc:2466] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.133206 1038 leader_election.cc:304] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.133512 1301 raft_consensus.cc:2802] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:29.133658 1301 raft_consensus.cc:491] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:29.133747 1301 raft_consensus.cc:3058] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.134142 1271 raft_consensus.cc:491] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.134296 1271 raft_consensus.cc:513] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.135001 1271 leader_election.cc:290] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.135741 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1032027a759344af84de0ca88d05566f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:29.135731 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1032027a759344af84de0ca88d05566f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:29.136022 1223 raft_consensus.cc:2466] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.136020 1150 raft_consensus.cc:2466] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.136452 1038 leader_election.cc:304] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.137202 1271 raft_consensus.cc:2802] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:29.137192 1301 raft_consensus.cc:513] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.137354 1271 raft_consensus.cc:491] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:29.137446 1271 raft_consensus.cc:3058] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.137976 1301 leader_election.cc:290] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.138408 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.138440 1150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:29.139153 1150 raft_consensus.cc:3058] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.139158 1223 raft_consensus.cc:3058] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.141839 1271 raft_consensus.cc:513] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.142810 1271 leader_election.cc:290] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.142472 1150 raft_consensus.cc:2466] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.143623 1149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1032027a759344af84de0ca88d05566f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:29.143620 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1032027a759344af84de0ca88d05566f" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.143909 1149 raft_consensus.cc:3058] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.143916 1224 raft_consensus.cc:3058] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.144641 1038 leader_election.cc:304] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters:
I20250626 01:51:29.145190 1271 raft_consensus.cc:2802] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:29.145370 1271 raft_consensus.cc:695] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.145714 1271 consensus_queue.cc:237] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.149581 971 catalog_manager.cc:5582] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.150085 1149 raft_consensus.cc:2466] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.150681 1038 leader_election.cc:304] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters:
I20250626 01:51:29.151044 1271 raft_consensus.cc:2802] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:29.151240 1271 raft_consensus.cc:695] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.151537 1271 consensus_queue.cc:237] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.154170 1223 raft_consensus.cc:2466] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.156908 1224 raft_consensus.cc:2466] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.160032 1271 raft_consensus.cc:491] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.160241 1271 raft_consensus.cc:513] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.161002 1271 leader_election.cc:290] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.161476 1149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:29.161710 1149 raft_consensus.cc:2466] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.162436 1038 leader_election.cc:304] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters:
I20250626 01:51:29.162734 1271 raft_consensus.cc:2802] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:29.162824 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:29.163084 1224 raft_consensus.cc:2466] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 0.
I20250626 01:51:29.162916 1271 raft_consensus.cc:491] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:29.163372 1271 raft_consensus.cc:3058] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.163168 971 catalog_manager.cc:5582] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.166253 1271 raft_consensus.cc:513] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.166945 1271 leader_election.cc:290] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Requested vote from peers 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.167701 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.167696 1149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7" candidate_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:29.167924 1224 raft_consensus.cc:3058] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.168017 1149 raft_consensus.cc:3058] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.170961 1224 raft_consensus.cc:2466] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.170961 1149 raft_consensus.cc:2466] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bff29e811ab4f89b8698bcc7553cc5a in term 1.
I20250626 01:51:29.171566 1038 leader_election.cc:304] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a; no voters:
I20250626 01:51:29.171901 1271 raft_consensus.cc:2802] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:29.172066 1271 raft_consensus.cc:695] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Leader. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.172376 1271 consensus_queue.cc:237] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.175809 971 catalog_manager.cc:5582] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.197664 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:29.200677 1319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:29.201086 1318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:29.201570 1321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:29.201686 413 server_base.cc:1048] running on GCE node
I20250626 01:51:29.202042 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:29.202133 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:29.202193 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902689202173 us; error 0 us; skew 500 ppm
I20250626 01:51:29.202384 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:29.203958 413 webserver.cc:469] Webserver started at http://127.0.103.68:42953/ using document root <none> and password file <none>
I20250626 01:51:29.204209 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:29.204300 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:29.204463 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:29.205121 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "813e33bf6ed645b7b3282044776442a0"
format_stamp: "Formatted at 2025-06-26 01:51:29 on dist-test-slave-psm5"
I20250626 01:51:29.207981 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:51:29.209720 1326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:29.210062 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:51:29.210175 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "813e33bf6ed645b7b3282044776442a0"
format_stamp: "Formatted at 2025-06-26 01:51:29 on dist-test-slave-psm5"
I20250626 01:51:29.210297 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:29.226438 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:29.227067 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:29.227638 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:29.228577 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:29.228672 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:29.228783 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:29.228856 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:29.248458 1388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:43487 every 8 connection(s)
I20250626 01:51:29.248450 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:43487
I20250626 01:51:29.256776 1389 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41023
I20250626 01:51:29.256953 1389 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:29.257483 1389 heartbeater.cc:507] Master 127.0.103.126:41023 requested a full tablet report, sending...
I20250626 01:51:29.258467 971 ts_manager.cc:194] Registered new tserver with Master: 813e33bf6ed645b7b3282044776442a0 (127.0.103.68:43487)
I20250626 01:51:29.259310 971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39600
I20250626 01:51:29.260188 1279 raft_consensus.cc:491] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.260368 1279 raft_consensus.cc:513] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.260983 1279 leader_election.cc:290] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771)
I20250626 01:51:29.261447 1149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74bdae8a7f4a436cabd708d053559461" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37a6f624375545bd81ce4d8d11635112" is_pre_election: true
I20250626 01:51:29.261489 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74bdae8a7f4a436cabd708d053559461" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" is_pre_election: true
I20250626 01:51:29.261673 1149 raft_consensus.cc:2391] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a050999cf24a4c7dbe77f664a561b6c3 in current term 1: Already voted for candidate 6bff29e811ab4f89b8698bcc7553cc5a in this term.
I20250626 01:51:29.262012 1187 leader_election.cc:304] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a050999cf24a4c7dbe77f664a561b6c3; no voters: 37a6f624375545bd81ce4d8d11635112, 6bff29e811ab4f89b8698bcc7553cc5a
I20250626 01:51:29.262269 1279 raft_consensus.cc:3058] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.264387 1279 raft_consensus.cc:2747] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 01:51:29.287508 1275 raft_consensus.cc:491] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:29.287753 1275 raft_consensus.cc:513] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } }
I20250626 01:51:29.288728 1275 leader_election.cc:290] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159), 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203)
I20250626 01:51:29.297156 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3039c4c5043945c29d4a559f79defe08" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" is_pre_election: true
I20250626 01:51:29.297633 1075 raft_consensus.cc:2391] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 37a6f624375545bd81ce4d8d11635112 in current term 1: Already voted for candidate a050999cf24a4c7dbe77f664a561b6c3 in this term.
I20250626 01:51:29.297668 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3039c4c5043945c29d4a559f79defe08" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3" is_pre_election: true
I20250626 01:51:29.298286 1113 leader_election.cc:304] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112; no voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3
I20250626 01:51:29.298658 1275 raft_consensus.cc:3058] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:29.300683 1275 raft_consensus.cc:2747] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 01:51:29.397665 1301 consensus_queue.cc:1035] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.402884 1301 consensus_queue.cc:1035] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.417598 1026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:29.426949 1279 consensus_queue.cc:1035] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.427727 1075 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "1032027a759344af84de0ca88d05566f"
dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
mode: GRACEFUL
new_leader_uuid: "37a6f624375545bd81ce4d8d11635112"
from {username='slave'} at 127.0.0.1:50864
I20250626 01:51:29.428045 1075 raft_consensus.cc:604] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Received request to transfer leadership to TS 37a6f624375545bd81ce4d8d11635112
I20250626 01:51:29.429257 1075 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7"
dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
mode: GRACEFUL
new_leader_uuid: "37a6f624375545bd81ce4d8d11635112"
from {username='slave'} at 127.0.0.1:50864
I20250626 01:51:29.429486 1075 raft_consensus.cc:604] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Received request to transfer leadership to TS 37a6f624375545bd81ce4d8d11635112
I20250626 01:51:29.431457 1075 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f"
dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
mode: GRACEFUL
new_leader_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
from {username='slave'} at 127.0.0.1:50864
I20250626 01:51:29.431692 1075 raft_consensus.cc:604] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Received request to transfer leadership to TS a050999cf24a4c7dbe77f664a561b6c3
I20250626 01:51:29.432128 1298 consensus_queue.cc:1035] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.433028 1075 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8"
dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
mode: GRACEFUL
new_leader_uuid: "37a6f624375545bd81ce4d8d11635112"
from {username='slave'} at 127.0.0.1:50864
I20250626 01:51:29.433248 1075 raft_consensus.cc:604] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Received request to transfer leadership to TS 37a6f624375545bd81ce4d8d11635112
I20250626 01:51:29.434525 1026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 01:51:29.434895 1026 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:29.442086 1301 consensus_queue.cc:1035] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.446633 1306 consensus_queue.cc:1035] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.464314 1271 consensus_queue.cc:1035] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.468485 1301 consensus_queue.cc:1035] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.495015 1301 consensus_queue.cc:1035] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.499104 1301 consensus_queue.cc:1035] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.738523 1301 consensus_queue.cc:1035] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.741408 1304 raft_consensus.cc:991] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: : Instructing follower 37a6f624375545bd81ce4d8d11635112 to start an election
I20250626 01:51:29.741611 1304 raft_consensus.cc:1079] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Signalling peer 37a6f624375545bd81ce4d8d11635112 to start an election
I20250626 01:51:29.742527 1150 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "1032027a759344af84de0ca88d05566f"
dest_uuid: "37a6f624375545bd81ce4d8d11635112"
from {username='slave'} at 127.0.0.1:49144
I20250626 01:51:29.742800 1150 raft_consensus.cc:491] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:51:29.742951 1150 raft_consensus.cc:3058] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.744289 1301 consensus_queue.cc:1035] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.746798 1150 raft_consensus.cc:513] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.747694 1150 leader_election.cc:290] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 2 election: Requested vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.748214 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1032027a759344af84de0ca88d05566f" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
I20250626 01:51:29.748224 1224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1032027a759344af84de0ca88d05566f" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.748466 1224 raft_consensus.cc:3058] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.748471 1075 raft_consensus.cc:3053] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:51:29.748620 1075 raft_consensus.cc:738] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.748952 1301 consensus_queue.cc:1035] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.749011 1075 consensus_queue.cc:260] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.749382 1075 raft_consensus.cc:3058] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.751361 1224 raft_consensus.cc:2466] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a6f624375545bd81ce4d8d11635112 in term 2.
I20250626 01:51:29.751785 1113 leader_election.cc:304] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.752115 1275 raft_consensus.cc:2802] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:51:29.752156 1075 raft_consensus.cc:2466] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a6f624375545bd81ce4d8d11635112 in term 2.
I20250626 01:51:29.753023 1275 raft_consensus.cc:695] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Becoming Leader. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Running, Role: LEADER
I20250626 01:51:29.753290 1301 raft_consensus.cc:991] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: : Instructing follower a050999cf24a4c7dbe77f664a561b6c3 to start an election
I20250626 01:51:29.753453 1301 raft_consensus.cc:1079] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Signalling peer a050999cf24a4c7dbe77f664a561b6c3 to start an election
I20250626 01:51:29.753350 1275 consensus_queue.cc:237] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.754562 1224 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f"
dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
from {username='slave'} at 127.0.0.1:47584
I20250626 01:51:29.754796 1224 raft_consensus.cc:491] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:51:29.754911 1224 raft_consensus.cc:3058] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.755424 1306 consensus_queue.cc:1035] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.756531 972 catalog_manager.cc:5582] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 reported cstate change: term changed from 1 to 2, leader changed from 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65) to 37a6f624375545bd81ce4d8d11635112 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "37a6f624375545bd81ce4d8d11635112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.760219 1224 raft_consensus.cc:513] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.761380 1224 leader_election.cc:290] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 2 election: Requested vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), 37a6f624375545bd81ce4d8d11635112 (127.0.103.66:33771)
I20250626 01:51:29.761528 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
I20250626 01:51:29.761648 1149 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4964c45e0491413c9baf98039bf2557f" candidate_uuid: "a050999cf24a4c7dbe77f664a561b6c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "37a6f624375545bd81ce4d8d11635112"
I20250626 01:51:29.761729 1075 raft_consensus.cc:3053] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:51:29.761850 1075 raft_consensus.cc:738] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.761858 1149 raft_consensus.cc:3058] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.762171 1075 consensus_queue.cc:260] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.762495 1075 raft_consensus.cc:3058] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.763305 1271 consensus_queue.cc:1035] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.765153 1149 raft_consensus.cc:2466] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050999cf24a4c7dbe77f664a561b6c3 in term 2.
I20250626 01:51:29.766192 1075 raft_consensus.cc:2466] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a050999cf24a4c7dbe77f664a561b6c3 in term 2.
I20250626 01:51:29.767609 1189 leader_election.cc:304] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bff29e811ab4f89b8698bcc7553cc5a, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.768080 1298 raft_consensus.cc:2802] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:51:29.768282 1298 raft_consensus.cc:695] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 LEADER]: Becoming Leader. State: Replica: a050999cf24a4c7dbe77f664a561b6c3, State: Running, Role: LEADER
I20250626 01:51:29.768589 1298 consensus_queue.cc:237] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.769630 1271 raft_consensus.cc:991] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: : Instructing follower 37a6f624375545bd81ce4d8d11635112 to start an election
I20250626 01:51:29.769800 1271 raft_consensus.cc:1079] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Signalling peer 37a6f624375545bd81ce4d8d11635112 to start an election
I20250626 01:51:29.771238 972 catalog_manager.cc:5582] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 reported cstate change: term changed from 1 to 2, leader changed from 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65) to a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "a050999cf24a4c7dbe77f664a561b6c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:29.771838 1149 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7"
dest_uuid: "37a6f624375545bd81ce4d8d11635112"
from {username='slave'} at 127.0.0.1:49144
I20250626 01:51:29.771870 1271 consensus_queue.cc:1035] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:29.773191 1149 raft_consensus.cc:491] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:51:29.773411 1149 raft_consensus.cc:3058] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.776465 1149 raft_consensus.cc:513] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.777997 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
I20250626 01:51:29.778240 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29908d5bcce84f7499b9a269e8f6e6f7" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.778424 1223 raft_consensus.cc:3058] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.778923 1149 leader_election.cc:290] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 2 election: Requested vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.779206 1075 raft_consensus.cc:3053] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:51:29.779366 1075 raft_consensus.cc:738] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.779654 1075 consensus_queue.cc:260] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.780035 1075 raft_consensus.cc:3058] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.780961 1223 raft_consensus.cc:2466] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a6f624375545bd81ce4d8d11635112 in term 2.
I20250626 01:51:29.781416 1113 leader_election.cc:304] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.781723 1275 raft_consensus.cc:2802] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:51:29.781893 1275 raft_consensus.cc:695] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Becoming Leader. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Running, Role: LEADER
I20250626 01:51:29.782171 1275 consensus_queue.cc:237] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.782632 1075 raft_consensus.cc:2466] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a6f624375545bd81ce4d8d11635112 in term 2.
I20250626 01:51:29.784929 971 catalog_manager.cc:5582] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 reported cstate change: term changed from 1 to 2, leader changed from 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65) to 37a6f624375545bd81ce4d8d11635112 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "37a6f624375545bd81ce4d8d11635112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:29.904376 1304 raft_consensus.cc:991] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: : Instructing follower 37a6f624375545bd81ce4d8d11635112 to start an election
I20250626 01:51:29.904553 1304 raft_consensus.cc:1079] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Signalling peer 37a6f624375545bd81ce4d8d11635112 to start an election
I20250626 01:51:29.905213 1149 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8"
dest_uuid: "37a6f624375545bd81ce4d8d11635112"
from {username='slave'} at 127.0.0.1:49144
I20250626 01:51:29.905411 1149 raft_consensus.cc:491] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:51:29.905990 1149 raft_consensus.cc:3058] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.907871 1149 raft_consensus.cc:513] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.908609 1149 leader_election.cc:290] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 2 election: Requested vote from peers 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65:33203), a050999cf24a4c7dbe77f664a561b6c3 (127.0.103.67:38159)
I20250626 01:51:29.909085 1223 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a050999cf24a4c7dbe77f664a561b6c3"
I20250626 01:51:29.909078 1075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e65bf8cdda3438da969f92e5b4a64b8" candidate_uuid: "37a6f624375545bd81ce4d8d11635112" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6bff29e811ab4f89b8698bcc7553cc5a"
I20250626 01:51:29.909345 1223 raft_consensus.cc:3058] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.909386 1075 raft_consensus.cc:3053] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:51:29.909469 1075 raft_consensus.cc:738] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6bff29e811ab4f89b8698bcc7553cc5a, State: Running, Role: LEADER
I20250626 01:51:29.909756 1075 consensus_queue.cc:260] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.910123 1075 raft_consensus.cc:3058] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:29.912675 1223 raft_consensus.cc:2466] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a6f624375545bd81ce4d8d11635112 in term 2.
I20250626 01:51:29.913103 1075 raft_consensus.cc:2466] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37a6f624375545bd81ce4d8d11635112 in term 2.
I20250626 01:51:29.913105 1113 leader_election.cc:304] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37a6f624375545bd81ce4d8d11635112, a050999cf24a4c7dbe77f664a561b6c3; no voters:
I20250626 01:51:29.913450 1275 raft_consensus.cc:2802] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:51:29.913640 1275 raft_consensus.cc:695] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Becoming Leader. State: Replica: 37a6f624375545bd81ce4d8d11635112, State: Running, Role: LEADER
I20250626 01:51:29.913899 1275 consensus_queue.cc:237] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } }
I20250626 01:51:29.915961 971 catalog_manager.cc:5582] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 reported cstate change: term changed from 1 to 2, leader changed from 6bff29e811ab4f89b8698bcc7553cc5a (127.0.103.65) to 37a6f624375545bd81ce4d8d11635112 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "37a6f624375545bd81ce4d8d11635112" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:30.209621 1075 raft_consensus.cc:1273] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.210224 1275 consensus_queue.cc:1035] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.213478 1224 raft_consensus.cc:1273] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.214252 1275 consensus_queue.cc:1035] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.260783 1389 heartbeater.cc:499] Master 127.0.103.126:41023 was elected leader, sending a full tablet report...
I20250626 01:51:30.303049 1149 raft_consensus.cc:1273] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Refusing update from remote peer a050999cf24a4c7dbe77f664a561b6c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.303769 1297 consensus_queue.cc:1035] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.307214 1075 raft_consensus.cc:1273] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Refusing update from remote peer a050999cf24a4c7dbe77f664a561b6c3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.308530 1298 consensus_queue.cc:1035] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.349221 1224 raft_consensus.cc:1273] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.349830 1275 consensus_queue.cc:1035] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.352855 1075 raft_consensus.cc:1273] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.353533 1419 consensus_queue.cc:1035] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.437031 1075 raft_consensus.cc:1273] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.437644 1419 consensus_queue.cc:1035] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:30.441522 1224 raft_consensus.cc:1273] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:51:30.442134 1275 consensus_queue.cc:1035] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:31.435451 1026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:31.437057 1026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:31.437247 1026 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:33.437762 1026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:33.439096 1026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:33.439287 1026 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:34.433691 1149 consensus_queue.cc:237] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } attrs { replace: true } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } }
I20250626 01:51:34.437074 1075 raft_consensus.cc:1273] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:51:34.437740 1224 raft_consensus.cc:1273] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:51:34.437913 1426 consensus_queue.cc:1035] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:34.438364 1427 consensus_queue.cc:1035] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:34.442204 1426 raft_consensus.cc:2953] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } attrs { replace: true } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.442945 1075 raft_consensus.cc:2953] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } attrs { replace: true } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.445207 1224 raft_consensus.cc:2953] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } attrs { replace: true } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
W20250626 01:51:34.448468 1113 consensus_peers.cc:487] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 -> Peer 813e33bf6ed645b7b3282044776442a0 (127.0.103.68:43487): Couldn't send request to peer 813e33bf6ed645b7b3282044776442a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1032027a759344af84de0ca88d05566f. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:51:34.448906 972 catalog_manager.cc:5582] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "37a6f624375545bd81ce4d8d11635112" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } attrs { replace: true } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.458974 1075 consensus_queue.cc:237] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } attrs { replace: true } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } }
I20250626 01:51:34.465785 1224 raft_consensus.cc:1273] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Refusing update from remote peer 6bff29e811ab4f89b8698bcc7553cc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:34.465785 1149 raft_consensus.cc:1273] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Refusing update from remote peer 6bff29e811ab4f89b8698bcc7553cc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:34.467103 1422 consensus_queue.cc:1035] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:34.467943 1422 consensus_queue.cc:1035] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 01:51:34.472560 1038 consensus_peers.cc:487] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a -> Peer 813e33bf6ed645b7b3282044776442a0 (127.0.103.68:43487): Couldn't send request to peer 813e33bf6ed645b7b3282044776442a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60acdfbdcf5b46fd885cbbe77992b763. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:51:34.474340 1439 raft_consensus.cc:2953] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } attrs { replace: true } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.474850 1224 raft_consensus.cc:2953] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } attrs { replace: true } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.475052 1149 raft_consensus.cc:2953] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } attrs { replace: true } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.479707 971 catalog_manager.cc:5582] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:34.481158 1075 consensus_queue.cc:237] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } }
W20250626 01:51:34.483737 1038 consensus_peers.cc:487] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a -> Peer 813e33bf6ed645b7b3282044776442a0 (127.0.103.68:43487): Couldn't send request to peer 813e33bf6ed645b7b3282044776442a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74bdae8a7f4a436cabd708d053559461. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:51:34.485659 1149 raft_consensus.cc:1273] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Refusing update from remote peer 6bff29e811ab4f89b8698bcc7553cc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:34.485868 1224 raft_consensus.cc:1273] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Refusing update from remote peer 6bff29e811ab4f89b8698bcc7553cc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:34.486299 1422 consensus_queue.cc:1035] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:34.486812 1439 consensus_queue.cc:1035] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:34.489415 1422 raft_consensus.cc:2953] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.490166 1224 raft_consensus.cc:2953] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.490166 1149 raft_consensus.cc:2953] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.494587 972 catalog_manager.cc:5582] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a reported cstate change: config changed from index -1 to 2, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:34.494773 1150 consensus_queue.cc:237] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } }
I20250626 01:51:34.498757 1075 raft_consensus.cc:1273] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250626 01:51:34.499394 1113 consensus_peers.cc:487] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 -> Peer 813e33bf6ed645b7b3282044776442a0 (127.0.103.68:43487): Couldn't send request to peer 813e33bf6ed645b7b3282044776442a0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e65bf8cdda3438da969f92e5b4a64b8. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:51:34.499802 1427 consensus_queue.cc:1035] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:34.501123 1224 raft_consensus.cc:1273] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Refusing update from remote peer 37a6f624375545bd81ce4d8d11635112: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:51:34.501619 1427 consensus_queue.cc:1035] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:34.503347 1426 raft_consensus.cc:2953] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.504108 1075 raft_consensus.cc:2953] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.505033 1224 raft_consensus.cc:2953] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } } }
I20250626 01:51:34.508208 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:51:34.509106 973 catalog_manager.cc:5582] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 reported cstate change: config changed from index -1 to 3, NON_VOTER 813e33bf6ed645b7b3282044776442a0 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "37a6f624375545bd81ce4d8d11635112" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bff29e811ab4f89b8698bcc7553cc5a" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37a6f624375545bd81ce4d8d11635112" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a050999cf24a4c7dbe77f664a561b6c3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38159 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "813e33bf6ed645b7b3282044776442a0" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 43487 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:51:34.511606 959 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20250626 01:51:34.513288 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:51:34.513442 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:51:34.519273 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:51:34.519428 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 01:51:34.520224 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:34.520486 413 tablet_replica.cc:331] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.520692 413 raft_consensus.cc:2241] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.520895 413 raft_consensus.cc:2270] T 4964c45e0491413c9baf98039bf2557f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.521512 413 tablet_replica.cc:331] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.521697 413 raft_consensus.cc:2241] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.521871 413 raft_consensus.cc:2270] T 29908d5bcce84f7499b9a269e8f6e6f7 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.522538 413 tablet_replica.cc:331] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.522733 413 raft_consensus.cc:2241] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.523082 413 raft_consensus.cc:2270] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.523682 413 tablet_replica.cc:331] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.523914 413 raft_consensus.cc:2241] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.524183 413 raft_consensus.cc:2270] T 6e65bf8cdda3438da969f92e5b4a64b8 P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.525008 413 tablet_replica.cc:331] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.525182 413 raft_consensus.cc:2241] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.525586 413 raft_consensus.cc:2270] T 60acdfbdcf5b46fd885cbbe77992b763 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.526191 413 tablet_replica.cc:331] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.526387 413 raft_consensus.cc:2241] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.526554 413 raft_consensus.cc:2270] T 3039c4c5043945c29d4a559f79defe08 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.527055 413 tablet_replica.cc:331] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.527207 413 raft_consensus.cc:2241] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.527441 413 raft_consensus.cc:2270] T 1032027a759344af84de0ca88d05566f P 6bff29e811ab4f89b8698bcc7553cc5a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.528124 413 tablet_replica.cc:331] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a: stopping tablet replica
I20250626 01:51:34.528285 413 raft_consensus.cc:2241] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.528620 413 raft_consensus.cc:2270] T 74bdae8a7f4a436cabd708d053559461 P 6bff29e811ab4f89b8698bcc7553cc5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.545411 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:51:34.550354 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
W20250626 01:51:34.553753 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:51:34.553929 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:51:34.556093 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:51:34.556252 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250626 01:51:34.561106 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:34.561414 413 tablet_replica.cc:331] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.561623 413 raft_consensus.cc:2241] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.561846 413 raft_consensus.cc:2270] T 60acdfbdcf5b46fd885cbbe77992b763 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.562723 413 tablet_replica.cc:331] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.562911 413 raft_consensus.cc:2241] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.563128 413 raft_consensus.cc:2270] T 3039c4c5043945c29d4a559f79defe08 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.563628 413 tablet_replica.cc:331] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.563793 413 raft_consensus.cc:2241] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.563966 413 raft_consensus.cc:2270] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.564417 413 tablet_replica.cc:331] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.564554 413 raft_consensus.cc:2241] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.564841 413 raft_consensus.cc:2270] T 29908d5bcce84f7499b9a269e8f6e6f7 P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.565249 413 tablet_replica.cc:331] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.565410 413 raft_consensus.cc:2241] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.565721 413 raft_consensus.cc:2270] T 6e65bf8cdda3438da969f92e5b4a64b8 P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.566574 413 tablet_replica.cc:331] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.566725 413 raft_consensus.cc:2241] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.566900 413 raft_consensus.cc:2270] T 74bdae8a7f4a436cabd708d053559461 P 37a6f624375545bd81ce4d8d11635112 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.567534 413 tablet_replica.cc:331] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.567684 413 raft_consensus.cc:2241] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.568045 413 raft_consensus.cc:2270] T 1032027a759344af84de0ca88d05566f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.568626 413 tablet_replica.cc:331] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112: stopping tablet replica
I20250626 01:51:34.568759 413 raft_consensus.cc:2241] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.568917 413 raft_consensus.cc:2270] T 4964c45e0491413c9baf98039bf2557f P 37a6f624375545bd81ce4d8d11635112 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.584550 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:51:34.588584 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:51:34.597416 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:34.597736 413 tablet_replica.cc:331] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.597955 413 raft_consensus.cc:2241] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.598210 413 raft_consensus.cc:2270] T 74bdae8a7f4a436cabd708d053559461 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.598976 413 tablet_replica.cc:331] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.599159 413 raft_consensus.cc:2241] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.599341 413 raft_consensus.cc:2270] T 1032027a759344af84de0ca88d05566f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.600195 413 tablet_replica.cc:331] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.600378 413 raft_consensus.cc:2241] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.600708 413 raft_consensus.cc:2270] T 4964c45e0491413c9baf98039bf2557f P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.601292 413 tablet_replica.cc:331] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.601465 413 raft_consensus.cc:2241] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.601625 413 raft_consensus.cc:2270] T 29908d5bcce84f7499b9a269e8f6e6f7 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.602120 413 tablet_replica.cc:331] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.602283 413 raft_consensus.cc:2241] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.602450 413 raft_consensus.cc:2270] T 39fd63d1d7354ab09ae3c1845ba3a4a9 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.602917 413 tablet_replica.cc:331] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.603088 413 raft_consensus.cc:2241] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.603379 413 raft_consensus.cc:2270] T 3039c4c5043945c29d4a559f79defe08 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.603857 413 tablet_replica.cc:331] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.604058 413 raft_consensus.cc:2241] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.604262 413 raft_consensus.cc:2270] T 60acdfbdcf5b46fd885cbbe77992b763 P a050999cf24a4c7dbe77f664a561b6c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.604983 413 tablet_replica.cc:331] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3: stopping tablet replica
I20250626 01:51:34.605150 413 raft_consensus.cc:2241] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:34.605335 413 raft_consensus.cc:2270] T 6e65bf8cdda3438da969f92e5b4a64b8 P a050999cf24a4c7dbe77f664a561b6c3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.620762 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:51:34.625077 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:51:34.633349 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:34.646339 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:51:34.649133 413 master.cc:561] Master@127.0.103.126:41023 shutting down...
I20250626 01:51:34.656636 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:34.656924 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:34.657055 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 96a70ca2c9b64ecc8846e1a1de62594f: stopping tablet replica
I20250626 01:51:34.661147 413 master.cc:583] Master@127.0.103.126:41023 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7332 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250626 01:51:34.677911 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:45987,127.0.103.125:40523,127.0.103.124:41709
I20250626 01:51:34.678526 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:34.681229 1451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.681239 1450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.681838 1453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:34.684993 413 server_base.cc:1048] running on GCE node
I20250626 01:51:34.686312 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:34.686403 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:34.686441 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902694686426 us; error 0 us; skew 500 ppm
I20250626 01:51:34.686582 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:34.688094 413 webserver.cc:469] Webserver started at http://127.0.103.126:34025/ using document root <none> and password file <none>
I20250626 01:51:34.688326 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:34.688416 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:34.688540 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:34.689198 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "b37b707062304065869120d8d8b970c4"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.691823 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:51:34.693603 1458 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.693995 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:51:34.694110 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "b37b707062304065869120d8d8b970c4"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.694267 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:34.707665 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:34.708415 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:34.722662 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:45987
I20250626 01:51:34.722685 1509 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:45987 every 8 connection(s)
I20250626 01:51:34.724169 1510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:34.724290 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:34.726944 1513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.727244 1512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.728104 1515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:34.728652 1510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:34.728861 413 server_base.cc:1048] running on GCE node
I20250626 01:51:34.729184 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:34.729281 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:34.729378 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902694729333 us; error 0 us; skew 500 ppm
I20250626 01:51:34.729635 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:34.731299 413 webserver.cc:469] Webserver started at http://127.0.103.125:37383/ using document root <none> and password file <none>
I20250626 01:51:34.731598 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:34.731746 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:34.731953 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:34.732846 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root/instance:
uuid: "202cf968158e4847b8a8f82d99f9aeff"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.736163 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:51:34.736898 1510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:34.738679 1522 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.739101 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250626 01:51:34.739207 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root
uuid: "202cf968158e4847b8a8f82d99f9aeff"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
W20250626 01:51:34.739303 1510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.125:40523: Network error: Client connection negotiation failed: client connection to 127.0.103.125:40523: connect: Connection refused (error 111)
I20250626 01:51:34.739372 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:34.749130 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:34.749800 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:34.761818 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:40523
I20250626 01:51:34.761855 1574 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.125:40523 every 8 connection(s)
I20250626 01:51:34.763211 1575 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:34.763290 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:34.765931 1577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.766230 1578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:34.767659 1575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } has no permanent_uuid. Determining permanent_uuid...
W20250626 01:51:34.767935 1580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:34.768275 413 server_base.cc:1048] running on GCE node
I20250626 01:51:34.768600 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:34.768733 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:34.768858 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902694768800 us; error 0 us; skew 500 ppm
I20250626 01:51:34.769053 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:34.770582 413 webserver.cc:469] Webserver started at http://127.0.103.124:33961/ using document root <none> and password file <none>
I20250626 01:51:34.770884 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:34.771009 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:34.771167 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:34.772003 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root/instance:
uuid: "6f09e5334d0a40bfad02358da62be0c2"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.774820 1575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:34.775332 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:51:34.777606 1587 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.778033 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:51:34.778153 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root
uuid: "6f09e5334d0a40bfad02358da62be0c2"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.778306 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:34.782536 1575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } has no permanent_uuid. Determining permanent_uuid...
W20250626 01:51:34.784545 1575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:41709: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:34.787550 1510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } attempt: 1
I20250626 01:51:34.793732 1510 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } has no permanent_uuid. Determining permanent_uuid...
W20250626 01:51:34.795487 1510 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.103.124:41709: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:34.809388 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:34.810004 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:34.823714 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:41709
I20250626 01:51:34.823752 1639 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.124:41709 every 8 connection(s)
I20250626 01:51:34.824733 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:51:34.825253 1640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:34.827621 1640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:34.831820 1575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } attempt: 1
I20250626 01:51:34.835719 1640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:34.842680 1640 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } has no permanent_uuid. Determining permanent_uuid...
I20250626 01:51:34.843221 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: Bootstrap starting.
I20250626 01:51:34.844940 1575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:34.846693 1510 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } attempt: 1
I20250626 01:51:34.846891 1575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: No bootstrap required, opened a new log
I20250626 01:51:34.848304 1575 raft_consensus.cc:357] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.848580 1575 raft_consensus.cc:383] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:34.848675 1575 raft_consensus.cc:738] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 202cf968158e4847b8a8f82d99f9aeff, State: Initialized, Role: FOLLOWER
I20250626 01:51:34.849035 1575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.854635 1640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2: Bootstrap starting.
I20250626 01:51:34.856109 1510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: Bootstrap starting.
I20250626 01:51:34.856045 1646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.856365 1646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:34.857409 1575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:34.857849 1510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:34.857463 1640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:34.860589 1510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: No bootstrap required, opened a new log
I20250626 01:51:34.861501 1510 raft_consensus.cc:357] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.861678 1510 raft_consensus.cc:383] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:34.861737 1510 raft_consensus.cc:738] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b37b707062304065869120d8d8b970c4, State: Initialized, Role: FOLLOWER
I20250626 01:51:34.861975 1510 consensus_queue.cc:260] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.862074 1640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2: No bootstrap required, opened a new log
I20250626 01:51:34.862767 1653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.863018 1510 sys_catalog.cc:564] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:34.863041 1653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:34.863651 1640 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.864048 1640 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:34.864171 1640 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f09e5334d0a40bfad02358da62be0c2, State: Initialized, Role: FOLLOWER
I20250626 01:51:34.864557 1640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.869274 1640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:34.869460 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
W20250626 01:51:34.870605 1665 catalog_manager.cc:1560] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 01:51:34.870733 1665 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 01:51:34.869350 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.870926 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:34.875036 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
W20250626 01:51:34.876262 1680 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 01:51:34.876387 1680 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250626 01:51:34.877319 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:34.877656 1683 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250626 01:51:34.877746 1683 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250626 01:51:34.880409 1685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.880666 1684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.880894 1687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:34.881436 413 server_base.cc:1048] running on GCE node
I20250626 01:51:34.881711 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:34.881783 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:34.881827 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902694881813 us; error 0 us; skew 500 ppm
I20250626 01:51:34.881966 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:34.883291 413 webserver.cc:469] Webserver started at http://127.0.103.65:35129/ using document root <none> and password file <none>
I20250626 01:51:34.883518 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:34.883586 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:34.883718 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:34.884932 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "f4c6de118d03434c92c8b517a4954e16"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.887667 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:51:34.889514 1692 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.889842 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:51:34.889973 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "f4c6de118d03434c92c8b517a4954e16"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.890106 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:34.903201 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:34.903838 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:34.904527 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:34.905580 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:34.905681 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.905786 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:34.905862 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.924544 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:33965
I20250626 01:51:34.924583 1754 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:33965 every 8 connection(s)
I20250626 01:51:34.929666 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:51:34.939595 1655 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:34.939899 1655 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.943641 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f09e5334d0a40bfad02358da62be0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202cf968158e4847b8a8f82d99f9aeff" is_pre_election: true
I20250626 01:51:34.944088 1757 heartbeater.cc:344] Connected to a master server at 127.0.103.125:40523
I20250626 01:51:34.944200 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f09e5334d0a40bfad02358da62be0c2 in term 0.
I20250626 01:51:34.944274 1757 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:34.941087 1655 leader_election.cc:290] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b37b707062304065869120d8d8b970c4 (127.0.103.126:45987), 202cf968158e4847b8a8f82d99f9aeff (127.0.103.125:40523)
I20250626 01:51:34.944734 1589 leader_election.cc:304] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff, 6f09e5334d0a40bfad02358da62be0c2; no voters:
I20250626 01:51:34.944965 1757 heartbeater.cc:507] Master 127.0.103.125:40523 requested a full tablet report, sending...
I20250626 01:51:34.944995 1485 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f09e5334d0a40bfad02358da62be0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37b707062304065869120d8d8b970c4" is_pre_election: true
I20250626 01:51:34.945791 1485 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f09e5334d0a40bfad02358da62be0c2 in term 0.
I20250626 01:51:34.946750 1540 ts_manager.cc:194] Registered new tserver with Master: f4c6de118d03434c92c8b517a4954e16 (127.0.103.65:33965)
I20250626 01:51:34.947006 1655 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:34.947175 1655 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:34.947340 1655 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 0 FOLLOWER]: Advancing to term 1
W20250626 01:51:34.947449 1765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.947477 1763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:34.949043 1762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:34.949486 1756 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45987
I20250626 01:51:34.949723 1756 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:34.949728 413 server_base.cc:1048] running on GCE node
I20250626 01:51:34.950209 1756 heartbeater.cc:507] Master 127.0.103.126:45987 requested a full tablet report, sending...
I20250626 01:51:34.950340 1755 heartbeater.cc:344] Connected to a master server at 127.0.103.124:41709
I20250626 01:51:34.950464 1755 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:34.950904 1755 heartbeater.cc:507] Master 127.0.103.124:41709 requested a full tablet report, sending...
I20250626 01:51:34.950953 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:34.951046 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:34.951121 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902694951080 us; error 0 us; skew 500 ppm
I20250626 01:51:34.951301 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:34.951378 1475 ts_manager.cc:194] Registered new tserver with Master: f4c6de118d03434c92c8b517a4954e16 (127.0.103.65:33965)
I20250626 01:51:34.951568 1655 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.952646 1605 ts_manager.cc:194] Registered new tserver with Master: f4c6de118d03434c92c8b517a4954e16 (127.0.103.65:33965)
I20250626 01:51:34.953096 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f09e5334d0a40bfad02358da62be0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "202cf968158e4847b8a8f82d99f9aeff"
I20250626 01:51:34.953329 1550 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:34.955308 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f09e5334d0a40bfad02358da62be0c2 in term 1.
I20250626 01:51:34.953121 1485 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6f09e5334d0a40bfad02358da62be0c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b37b707062304065869120d8d8b970c4"
I20250626 01:51:34.955632 1485 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:34.955927 1589 leader_election.cc:304] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff, 6f09e5334d0a40bfad02358da62be0c2; no voters:
I20250626 01:51:34.956750 1769 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:34.957234 1769 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 1 LEADER]: Becoming Leader. State: Replica: 6f09e5334d0a40bfad02358da62be0c2, State: Running, Role: LEADER
I20250626 01:51:34.957705 1769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:34.958580 1485 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f09e5334d0a40bfad02358da62be0c2 in term 1.
I20250626 01:51:34.958861 1655 leader_election.cc:290] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [CANDIDATE]: Term 1 election: Requested vote from peers b37b707062304065869120d8d8b970c4 (127.0.103.126:45987), 202cf968158e4847b8a8f82d99f9aeff (127.0.103.125:40523)
I20250626 01:51:34.959403 413 webserver.cc:469] Webserver started at http://127.0.103.66:43081/ using document root <none> and password file <none>
I20250626 01:51:34.959676 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:34.959852 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:34.960044 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:34.959805 1770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f09e5334d0a40bfad02358da62be0c2. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.960155 1770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:34.961354 1772 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:51:34.961331 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "a022a5be2edd47f68f8aa3e4e471dd1c"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.962805 1772 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:51:34.964358 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250626 01:51:34.966418 1550 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Refusing update from remote peer 6f09e5334d0a40bfad02358da62be0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:34.966715 1775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:34.967139 1485 raft_consensus.cc:1273] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Refusing update from remote peer 6f09e5334d0a40bfad02358da62be0c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:34.967087 1770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:34.967790 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250626 01:51:34.967751 1770 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:34.967990 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "a022a5be2edd47f68f8aa3e4e471dd1c"
format_stamp: "Formatted at 2025-06-26 01:51:34 on dist-test-slave-psm5"
I20250626 01:51:34.968134 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:34.978225 1646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f09e5334d0a40bfad02358da62be0c2. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.978618 1646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:34.983865 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.984222 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:34.986406 1653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f09e5334d0a40bfad02358da62be0c2. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.986692 1653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:34.986919 1772 catalog_manager.cc:1349] Generated new cluster ID: c9a75b826c8a4f949c2b1a6a34130294
I20250626 01:51:34.987015 1772 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:51:34.987545 1655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:34.987737 1655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:35.001766 1646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:35.002722 1646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:35.006103 1653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f09e5334d0a40bfad02358da62be0c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:35.006363 1653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:35.008780 1772 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:51:35.009829 1772 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:51:35.009269 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:35.012708 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:35.013403 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:35.014592 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:35.014683 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:35.014796 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:35.014858 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:35.027567 1772 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2: Generated new TSK 0
I20250626 01:51:35.028119 1772 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:51:35.038122 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:39515
I20250626 01:51:35.038161 1844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:39515 every 8 connection(s)
I20250626 01:51:35.049937 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:51:35.050441 1845 heartbeater.cc:344] Connected to a master server at 127.0.103.124:41709
I20250626 01:51:35.051075 1845 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:35.051697 1845 heartbeater.cc:507] Master 127.0.103.124:41709 requested a full tablet report, sending...
I20250626 01:51:35.052757 1846 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45987
I20250626 01:51:35.052933 1846 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:35.053014 1605 ts_manager.cc:194] Registered new tserver with Master: a022a5be2edd47f68f8aa3e4e471dd1c (127.0.103.66:39515)
I20250626 01:51:35.053468 1846 heartbeater.cc:507] Master 127.0.103.126:45987 requested a full tablet report, sending...
I20250626 01:51:35.055338 1605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40746
I20250626 01:51:35.056422 1475 ts_manager.cc:194] Registered new tserver with Master: a022a5be2edd47f68f8aa3e4e471dd1c (127.0.103.66:39515)
I20250626 01:51:35.057996 1847 heartbeater.cc:344] Connected to a master server at 127.0.103.125:40523
I20250626 01:51:35.058156 1847 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:35.058499 1847 heartbeater.cc:507] Master 127.0.103.125:40523 requested a full tablet report, sending...
W20250626 01:51:35.059311 1852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:35.060258 1540 ts_manager.cc:194] Registered new tserver with Master: a022a5be2edd47f68f8aa3e4e471dd1c (127.0.103.66:39515)
W20250626 01:51:35.061017 1853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:35.061872 1855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:35.065143 413 server_base.cc:1048] running on GCE node
I20250626 01:51:35.065435 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:35.065521 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:35.065582 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902695065561 us; error 0 us; skew 500 ppm
I20250626 01:51:35.065750 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:35.067278 413 webserver.cc:469] Webserver started at http://127.0.103.67:40439/ using document root <none> and password file <none>
I20250626 01:51:35.067525 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:35.067610 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:35.067762 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:35.068475 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "f18c1eaab20743b78793bd2cc49ad43e"
format_stamp: "Formatted at 2025-06-26 01:51:35 on dist-test-slave-psm5"
I20250626 01:51:35.071415 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250626 01:51:35.073660 1860 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:35.074102 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:51:35.074223 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "f18c1eaab20743b78793bd2cc49ad43e"
format_stamp: "Formatted at 2025-06-26 01:51:35 on dist-test-slave-psm5"
I20250626 01:51:35.074337 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:35.093403 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:35.094564 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:35.095242 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:35.096240 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:35.096339 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:35.096446 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:35.096522 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:35.114086 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:33077
I20250626 01:51:35.114135 1922 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:33077 every 8 connection(s)
I20250626 01:51:35.125949 1923 heartbeater.cc:344] Connected to a master server at 127.0.103.124:41709
I20250626 01:51:35.126155 1923 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:35.126686 1925 heartbeater.cc:344] Connected to a master server at 127.0.103.125:40523
I20250626 01:51:35.126747 1923 heartbeater.cc:507] Master 127.0.103.124:41709 requested a full tablet report, sending...
I20250626 01:51:35.126808 1925 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:35.127233 1925 heartbeater.cc:507] Master 127.0.103.125:40523 requested a full tablet report, sending...
I20250626 01:51:35.127741 1605 ts_manager.cc:194] Registered new tserver with Master: f18c1eaab20743b78793bd2cc49ad43e (127.0.103.67:33077)
I20250626 01:51:35.127815 1924 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45987
I20250626 01:51:35.127998 1924 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:35.128504 1540 ts_manager.cc:194] Registered new tserver with Master: f18c1eaab20743b78793bd2cc49ad43e (127.0.103.67:33077)
I20250626 01:51:35.128561 1924 heartbeater.cc:507] Master 127.0.103.126:45987 requested a full tablet report, sending...
I20250626 01:51:35.128778 1605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40748
I20250626 01:51:35.129534 1475 ts_manager.cc:194] Registered new tserver with Master: f18c1eaab20743b78793bd2cc49ad43e (127.0.103.67:33077)
I20250626 01:51:35.130504 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013576781s
I20250626 01:51:35.872283 1665 catalog_manager.cc:1261] Loaded cluster ID: c9a75b826c8a4f949c2b1a6a34130294
I20250626 01:51:35.872419 1665 catalog_manager.cc:1554] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: loading cluster ID for follower catalog manager: success
I20250626 01:51:35.874507 1665 catalog_manager.cc:1576] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: acquiring CA information for follower catalog manager: success
I20250626 01:51:35.875754 1665 catalog_manager.cc:1604] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 01:51:35.877637 1680 catalog_manager.cc:1261] Loaded cluster ID: c9a75b826c8a4f949c2b1a6a34130294
I20250626 01:51:35.877728 1680 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: loading cluster ID for follower catalog manager: success
I20250626 01:51:35.879644 1680 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: acquiring CA information for follower catalog manager: success
I20250626 01:51:35.880801 1680 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250626 01:51:35.955403 1605 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40738
I20250626 01:51:36.057574 1845 heartbeater.cc:499] Master 127.0.103.124:41709 was elected leader, sending a full tablet report...
I20250626 01:51:36.130102 1923 heartbeater.cc:499] Master 127.0.103.124:41709 was elected leader, sending a full tablet report...
I20250626 01:51:36.876598 1682 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:36.956740 1755 heartbeater.cc:499] Master 127.0.103.124:41709 was elected leader, sending a full tablet report...
I20250626 01:51:37.192232 413 test_util.cc:276] Using random seed: -850564335
I20250626 01:51:37.217140 1605 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54732:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 01:51:37.274036 1888 tablet_service.cc:1468] Processing CreateTablet for tablet 8b8000d319624343873e1c5e4f26cee3 (DEFAULT_TABLE table=test-workload [id=69ae62d55b7346e99585c35d3ebab4e4]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 01:51:37.274586 1888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b8000d319624343873e1c5e4f26cee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:37.275513 1720 tablet_service.cc:1468] Processing CreateTablet for tablet 20a4ca53dbae4127a0eea2bf2a16c121 (DEFAULT_TABLE table=test-workload [id=69ae62d55b7346e99585c35d3ebab4e4]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 01:51:37.276095 1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a4ca53dbae4127a0eea2bf2a16c121. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:37.279963 1810 tablet_service.cc:1468] Processing CreateTablet for tablet 8ae6b2c088944ddf853c07cdaf1fe4c1 (DEFAULT_TABLE table=test-workload [id=69ae62d55b7346e99585c35d3ebab4e4]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 01:51:37.280741 1810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ae6b2c088944ddf853c07cdaf1fe4c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:37.285601 1961 tablet_bootstrap.cc:492] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16: Bootstrap starting.
I20250626 01:51:37.288445 1961 tablet_bootstrap.cc:654] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:37.288638 1960 tablet_bootstrap.cc:492] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e: Bootstrap starting.
I20250626 01:51:37.289952 1962 tablet_bootstrap.cc:492] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c: Bootstrap starting.
I20250626 01:51:37.291532 1960 tablet_bootstrap.cc:654] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:37.291718 1961 tablet_bootstrap.cc:492] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16: No bootstrap required, opened a new log
I20250626 01:51:37.291949 1961 ts_tablet_manager.cc:1397] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250626 01:51:37.292351 1962 tablet_bootstrap.cc:654] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:37.292752 1961 raft_consensus.cc:357] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6de118d03434c92c8b517a4954e16" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33965 } }
I20250626 01:51:37.292994 1961 raft_consensus.cc:383] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:37.293941 1961 raft_consensus.cc:738] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c6de118d03434c92c8b517a4954e16, State: Initialized, Role: FOLLOWER
I20250626 01:51:37.294938 1960 tablet_bootstrap.cc:492] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e: No bootstrap required, opened a new log
I20250626 01:51:37.294863 1961 consensus_queue.cc:260] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6de118d03434c92c8b517a4954e16" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33965 } }
I20250626 01:51:37.295100 1960 ts_tablet_manager.cc:1397] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250626 01:51:37.295300 1961 raft_consensus.cc:397] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:37.295418 1961 raft_consensus.cc:491] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:37.295547 1961 raft_consensus.cc:3058] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:37.295724 1962 tablet_bootstrap.cc:492] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c: No bootstrap required, opened a new log
I20250626 01:51:37.295943 1962 ts_tablet_manager.cc:1397] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250626 01:51:37.296095 1960 raft_consensus.cc:357] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c1eaab20743b78793bd2cc49ad43e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33077 } }
I20250626 01:51:37.296319 1960 raft_consensus.cc:383] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:37.296443 1960 raft_consensus.cc:738] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f18c1eaab20743b78793bd2cc49ad43e, State: Initialized, Role: FOLLOWER
I20250626 01:51:37.296702 1960 consensus_queue.cc:260] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c1eaab20743b78793bd2cc49ad43e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33077 } }
I20250626 01:51:37.296818 1962 raft_consensus.cc:357] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a022a5be2edd47f68f8aa3e4e471dd1c" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39515 } }
I20250626 01:51:37.296921 1960 raft_consensus.cc:397] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:37.296976 1962 raft_consensus.cc:383] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:37.297061 1960 raft_consensus.cc:491] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:37.297125 1962 raft_consensus.cc:738] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a022a5be2edd47f68f8aa3e4e471dd1c, State: Initialized, Role: FOLLOWER
I20250626 01:51:37.297211 1960 raft_consensus.cc:3058] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:37.297384 1962 consensus_queue.cc:260] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a022a5be2edd47f68f8aa3e4e471dd1c" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39515 } }
I20250626 01:51:37.297610 1962 raft_consensus.cc:397] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:37.297739 1962 raft_consensus.cc:491] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:37.297891 1962 raft_consensus.cc:3058] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:37.298255 1961 raft_consensus.cc:513] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6de118d03434c92c8b517a4954e16" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33965 } }
I20250626 01:51:37.298506 1961 leader_election.cc:304] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4c6de118d03434c92c8b517a4954e16; no voters:
I20250626 01:51:37.299083 1961 leader_election.cc:290] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:37.299330 1966 raft_consensus.cc:2802] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:37.300436 1960 raft_consensus.cc:513] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c1eaab20743b78793bd2cc49ad43e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33077 } }
I20250626 01:51:37.300771 1960 leader_election.cc:304] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f18c1eaab20743b78793bd2cc49ad43e; no voters:
I20250626 01:51:37.301653 1962 raft_consensus.cc:513] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a022a5be2edd47f68f8aa3e4e471dd1c" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39515 } }
I20250626 01:51:37.301918 1962 leader_election.cc:304] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a022a5be2edd47f68f8aa3e4e471dd1c; no voters:
I20250626 01:51:37.303848 1969 raft_consensus.cc:2802] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:37.304386 1960 leader_election.cc:290] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:37.305763 1969 raft_consensus.cc:695] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 1 LEADER]: Becoming Leader. State: Replica: f18c1eaab20743b78793bd2cc49ad43e, State: Running, Role: LEADER
I20250626 01:51:37.306233 1960 ts_tablet_manager.cc:1428] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20250626 01:51:37.307546 1966 raft_consensus.cc:695] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 1 LEADER]: Becoming Leader. State: Replica: f4c6de118d03434c92c8b517a4954e16, State: Running, Role: LEADER
I20250626 01:51:37.306262 1969 consensus_queue.cc:237] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c1eaab20743b78793bd2cc49ad43e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33077 } }
I20250626 01:51:37.309017 1961 ts_tablet_manager.cc:1428] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16: Time spent starting tablet: real 0.016s user 0.006s sys 0.007s
I20250626 01:51:37.309101 1966 consensus_queue.cc:237] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6de118d03434c92c8b517a4954e16" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33965 } }
I20250626 01:51:37.311570 1962 leader_election.cc:290] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:37.312201 1970 raft_consensus.cc:2802] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:37.312865 1962 ts_tablet_manager.cc:1428] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c: Time spent starting tablet: real 0.017s user 0.007s sys 0.004s
I20250626 01:51:37.313113 1970 raft_consensus.cc:695] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 1 LEADER]: Becoming Leader. State: Replica: a022a5be2edd47f68f8aa3e4e471dd1c, State: Running, Role: LEADER
I20250626 01:51:37.313431 1970 consensus_queue.cc:237] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a022a5be2edd47f68f8aa3e4e471dd1c" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39515 } }
I20250626 01:51:37.316552 1605 catalog_manager.cc:5582] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c reported cstate change: term changed from 0 to 1, leader changed from <none> to a022a5be2edd47f68f8aa3e4e471dd1c (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a022a5be2edd47f68f8aa3e4e471dd1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a022a5be2edd47f68f8aa3e4e471dd1c" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39515 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:37.324656 1604 catalog_manager.cc:5582] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e reported cstate change: term changed from 0 to 1, leader changed from <none> to f18c1eaab20743b78793bd2cc49ad43e (127.0.103.67). New cstate: current_term: 1 leader_uuid: "f18c1eaab20743b78793bd2cc49ad43e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f18c1eaab20743b78793bd2cc49ad43e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33077 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:37.329851 1605 catalog_manager.cc:5582] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c6de118d03434c92c8b517a4954e16 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "f4c6de118d03434c92c8b517a4954e16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4c6de118d03434c92c8b517a4954e16" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33965 } health_report { overall_health: HEALTHY } } }
I20250626 01:51:37.357221 413 master.cc:561] Master@127.0.103.124:41709 shutting down...
I20250626 01:51:37.363976 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:37.364409 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:37.364539 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6f09e5334d0a40bfad02358da62be0c2: stopping tablet replica
I20250626 01:51:37.380482 413 master.cc:583] Master@127.0.103.124:41709 shutdown complete.
W20250626 01:51:38.348901 1755 heartbeater.cc:646] Failed to heartbeat to 127.0.103.124:41709 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:38.849799 1984 raft_consensus.cc:491] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f09e5334d0a40bfad02358da62be0c2)
I20250626 01:51:38.849869 1985 raft_consensus.cc:491] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6f09e5334d0a40bfad02358da62be0c2)
I20250626 01:51:38.850080 1984 raft_consensus.cc:513] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:38.850104 1985 raft_consensus.cc:513] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:38.850816 1985 leader_election.cc:290] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b37b707062304065869120d8d8b970c4 (127.0.103.126:45987), 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709)
I20250626 01:51:38.851369 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b37b707062304065869120d8d8b970c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "202cf968158e4847b8a8f82d99f9aeff" is_pre_election: true
I20250626 01:51:38.851662 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37b707062304065869120d8d8b970c4 in term 1.
I20250626 01:51:38.851858 1984 leader_election.cc:290] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 202cf968158e4847b8a8f82d99f9aeff (127.0.103.125:40523), 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709)
I20250626 01:51:38.851976 1485 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202cf968158e4847b8a8f82d99f9aeff" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "b37b707062304065869120d8d8b970c4" is_pre_election: true
I20250626 01:51:38.852180 1485 raft_consensus.cc:2466] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 202cf968158e4847b8a8f82d99f9aeff in term 1.
I20250626 01:51:38.852715 1460 leader_election.cc:304] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff, b37b707062304065869120d8d8b970c4; no voters:
I20250626 01:51:38.852823 1524 leader_election.cc:304] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff, b37b707062304065869120d8d8b970c4; no voters:
I20250626 01:51:38.852978 1984 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:51:38.853101 1984 raft_consensus.cc:491] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f09e5334d0a40bfad02358da62be0c2)
I20250626 01:51:38.853229 1984 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:51:38.853473 1985 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:51:38.853626 1985 raft_consensus.cc:491] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6f09e5334d0a40bfad02358da62be0c2)
I20250626 01:51:38.853751 1985 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 1 FOLLOWER]: Advancing to term 2
W20250626 01:51:38.854156 1459 leader_election.cc:336] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
W20250626 01:51:38.855244 1524 leader_election.cc:336] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:38.856010 1984 raft_consensus.cc:513] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:38.856611 1985 raft_consensus.cc:513] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:38.857117 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b37b707062304065869120d8d8b970c4" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "202cf968158e4847b8a8f82d99f9aeff"
I20250626 01:51:38.857234 1985 leader_election.cc:290] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [CANDIDATE]: Term 2 election: Requested vote from peers b37b707062304065869120d8d8b970c4 (127.0.103.126:45987), 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709)
I20250626 01:51:38.857395 1550 raft_consensus.cc:2391] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b37b707062304065869120d8d8b970c4 in current term 2: Already voted for candidate 202cf968158e4847b8a8f82d99f9aeff in this term.
I20250626 01:51:38.856619 1984 leader_election.cc:290] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 2 election: Requested vote from peers 202cf968158e4847b8a8f82d99f9aeff (127.0.103.125:40523), 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709)
I20250626 01:51:38.857743 1485 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "202cf968158e4847b8a8f82d99f9aeff" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "b37b707062304065869120d8d8b970c4"
I20250626 01:51:38.857997 1485 raft_consensus.cc:2391] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 202cf968158e4847b8a8f82d99f9aeff in current term 2: Already voted for candidate b37b707062304065869120d8d8b970c4 in this term.
W20250626 01:51:38.858175 1459 leader_election.cc:336] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:38.858330 1459 leader_election.cc:304] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b37b707062304065869120d8d8b970c4; no voters: 202cf968158e4847b8a8f82d99f9aeff, 6f09e5334d0a40bfad02358da62be0c2
I20250626 01:51:38.858655 1984 raft_consensus.cc:2747] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250626 01:51:38.859274 1524 leader_election.cc:336] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:38.859378 1524 leader_election.cc:304] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff; no voters: 6f09e5334d0a40bfad02358da62be0c2, b37b707062304065869120d8d8b970c4
I20250626 01:51:38.859580 1985 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250626 01:51:40.397119 1997 raft_consensus.cc:491] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:40.397325 1997 raft_consensus.cc:513] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:40.398020 1997 leader_election.cc:290] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 202cf968158e4847b8a8f82d99f9aeff (127.0.103.125:40523), 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709)
I20250626 01:51:40.398680 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b37b707062304065869120d8d8b970c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "202cf968158e4847b8a8f82d99f9aeff" is_pre_election: true
I20250626 01:51:40.398896 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b37b707062304065869120d8d8b970c4 in term 2.
W20250626 01:51:40.399030 1459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250626 01:51:40.399412 1460 leader_election.cc:304] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff, b37b707062304065869120d8d8b970c4; no voters:
I20250626 01:51:40.399749 1997 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250626 01:51:40.399873 1997 raft_consensus.cc:491] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:40.400002 1997 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 2 FOLLOWER]: Advancing to term 3
W20250626 01:51:40.400485 1459 leader_election.cc:336] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:40.401935 1997 raft_consensus.cc:513] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:40.402515 1997 leader_election.cc:290] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 3 election: Requested vote from peers 202cf968158e4847b8a8f82d99f9aeff (127.0.103.125:40523), 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709)
I20250626 01:51:40.403008 1550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b37b707062304065869120d8d8b970c4" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "202cf968158e4847b8a8f82d99f9aeff"
I20250626 01:51:40.403179 1550 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 2 FOLLOWER]: Advancing to term 3
W20250626 01:51:40.403872 1459 leader_election.cc:336] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111)
I20250626 01:51:40.405195 1550 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b37b707062304065869120d8d8b970c4 in term 3.
I20250626 01:51:40.405624 1460 leader_election.cc:304] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 202cf968158e4847b8a8f82d99f9aeff, b37b707062304065869120d8d8b970c4; no voters: 6f09e5334d0a40bfad02358da62be0c2
I20250626 01:51:40.405920 1997 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:51:40.406286 1997 raft_consensus.cc:695] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 3 LEADER]: Becoming Leader. State: Replica: b37b707062304065869120d8d8b970c4, State: Running, Role: LEADER
I20250626 01:51:40.406543 1997 consensus_queue.cc:237] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } }
I20250626 01:51:40.407764 2000 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37b707062304065869120d8d8b970c4. Latest consensus state: current_term: 3 leader_uuid: "b37b707062304065869120d8d8b970c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:40.408044 2000 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:40.408558 2002 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:51:40.410307 2002 catalog_manager.cc:671] Loaded metadata for table test-workload [id=69ae62d55b7346e99585c35d3ebab4e4]
I20250626 01:51:40.411752 2002 tablet_loader.cc:96] loaded metadata for tablet 20a4ca53dbae4127a0eea2bf2a16c121 (table test-workload [id=69ae62d55b7346e99585c35d3ebab4e4])
I20250626 01:51:40.412158 2002 tablet_loader.cc:96] loaded metadata for tablet 8ae6b2c088944ddf853c07cdaf1fe4c1 (table test-workload [id=69ae62d55b7346e99585c35d3ebab4e4])
I20250626 01:51:40.412483 2002 tablet_loader.cc:96] loaded metadata for tablet 8b8000d319624343873e1c5e4f26cee3 (table test-workload [id=69ae62d55b7346e99585c35d3ebab4e4])
I20250626 01:51:40.412735 2002 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:51:40.413633 2002 catalog_manager.cc:1261] Loaded cluster ID: c9a75b826c8a4f949c2b1a6a34130294
I20250626 01:51:40.413727 2002 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:51:40.414876 2002 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:51:40.415697 2002 catalog_manager.cc:5966] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: Loaded TSK: 0
I20250626 01:51:40.416392 2002 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:51:40.869397 1664 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:40.869537 1664 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:40.912950 1550 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 3 FOLLOWER]: Refusing update from remote peer b37b707062304065869120d8d8b970c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20250626 01:51:40.913615 2003 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250626 01:51:40.915390 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b37b707062304065869120d8d8b970c4. Latest consensus state: current_term: 3 leader_uuid: "b37b707062304065869120d8d8b970c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:40.915625 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: This master's current role is: FOLLOWER
I20250626 01:51:40.917258 2003 sys_catalog.cc:455] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b37b707062304065869120d8d8b970c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:40.917650 2003 sys_catalog.cc:458] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:40.917699 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b37b707062304065869120d8d8b970c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b37b707062304065869120d8d8b970c4" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45987 } } peers { permanent_uuid: "202cf968158e4847b8a8f82d99f9aeff" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 40523 } } peers { permanent_uuid: "6f09e5334d0a40bfad02358da62be0c2" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41709 } } }
I20250626 01:51:40.917946 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [sys.catalog]: This master's current role is: FOLLOWER
W20250626 01:51:40.919418 1459 consensus_peers.cc:487] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 -> Peer 6f09e5334d0a40bfad02358da62be0c2 (127.0.103.124:41709): Couldn't send request to peer 6f09e5334d0a40bfad02358da62be0c2. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41709: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:51:41.018225 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:51:41.027009 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:41.027351 413 tablet_replica.cc:331] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16: stopping tablet replica
I20250626 01:51:41.027561 413 raft_consensus.cc:2241] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:41.027743 413 raft_consensus.cc:2270] T 20a4ca53dbae4127a0eea2bf2a16c121 P f4c6de118d03434c92c8b517a4954e16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:41.041417 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:51:41.044413 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:51:41.053471 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:41.053762 413 tablet_replica.cc:331] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c: stopping tablet replica
I20250626 01:51:41.053961 413 raft_consensus.cc:2241] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:41.054140 413 raft_consensus.cc:2270] T 8ae6b2c088944ddf853c07cdaf1fe4c1 P a022a5be2edd47f68f8aa3e4e471dd1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:41.068069 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:51:41.071115 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:51:41.079975 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:51:41.080281 413 tablet_replica.cc:331] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e: stopping tablet replica
I20250626 01:51:41.080472 413 raft_consensus.cc:2241] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:51:41.080672 413 raft_consensus.cc:2270] T 8b8000d319624343873e1c5e4f26cee3 P f18c1eaab20743b78793bd2cc49ad43e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:41.094411 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:51:41.097471 413 master.cc:561] Master@127.0.103.126:45987 shutting down...
I20250626 01:51:41.105052 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 3 LEADER]: Raft consensus shutting down.
I20250626 01:51:41.105494 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:41.105713 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P b37b707062304065869120d8d8b970c4: stopping tablet replica
I20250626 01:51:41.110577 413 master.cc:583] Master@127.0.103.126:45987 shutdown complete.
I20250626 01:51:41.114476 413 master.cc:561] Master@127.0.103.125:40523 shutting down...
I20250626 01:51:41.121058 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:41.121325 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:41.121423 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 202cf968158e4847b8a8f82d99f9aeff: stopping tablet replica
I20250626 01:51:41.136049 413 master.cc:583] Master@127.0.103.125:40523 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6471 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250626 01:51:41.149545 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:34087
I20250626 01:51:41.150161 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:41.153455 2015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.153553 2012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.153513 2013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:41.153923 413 server_base.cc:1048] running on GCE node
I20250626 01:51:41.154249 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:41.154323 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:41.154368 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902701154348 us; error 0 us; skew 500 ppm
I20250626 01:51:41.154557 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:41.161998 413 webserver.cc:469] Webserver started at http://127.0.103.126:38305/ using document root <none> and password file <none>
I20250626 01:51:41.162317 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:41.162429 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:41.162602 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:41.163373 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "c9467a6ed89343f1845ff6a7581e6090"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.166399 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250626 01:51:41.168303 2020 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.168757 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:51:41.168866 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "c9467a6ed89343f1845ff6a7581e6090"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.169015 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:41.178957 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:41.179611 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:41.193228 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:34087
I20250626 01:51:41.193262 2071 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:34087 every 8 connection(s)
I20250626 01:51:41.194679 2072 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:41.198797 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090: Bootstrap starting.
I20250626 01:51:41.200258 2072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:41.201972 2072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090: No bootstrap required, opened a new log
I20250626 01:51:41.202646 2072 raft_consensus.cc:357] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9467a6ed89343f1845ff6a7581e6090" member_type: VOTER }
I20250626 01:51:41.202795 2072 raft_consensus.cc:383] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:41.202885 2072 raft_consensus.cc:738] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9467a6ed89343f1845ff6a7581e6090, State: Initialized, Role: FOLLOWER
I20250626 01:51:41.203099 2072 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9467a6ed89343f1845ff6a7581e6090" member_type: VOTER }
I20250626 01:51:41.203246 2072 raft_consensus.cc:397] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:51:41.203315 2072 raft_consensus.cc:491] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:51:41.203398 2072 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:41.205174 2072 raft_consensus.cc:513] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9467a6ed89343f1845ff6a7581e6090" member_type: VOTER }
I20250626 01:51:41.205403 2072 leader_election.cc:304] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9467a6ed89343f1845ff6a7581e6090; no voters:
I20250626 01:51:41.205803 2072 leader_election.cc:290] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:51:41.205987 2075 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:41.206490 2075 raft_consensus.cc:695] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 1 LEADER]: Becoming Leader. State: Replica: c9467a6ed89343f1845ff6a7581e6090, State: Running, Role: LEADER
I20250626 01:51:41.206784 2072 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:51:41.206756 2075 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9467a6ed89343f1845ff6a7581e6090" member_type: VOTER }
I20250626 01:51:41.207716 2076 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9467a6ed89343f1845ff6a7581e6090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9467a6ed89343f1845ff6a7581e6090" member_type: VOTER } }
I20250626 01:51:41.207783 2077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9467a6ed89343f1845ff6a7581e6090. Latest consensus state: current_term: 1 leader_uuid: "c9467a6ed89343f1845ff6a7581e6090" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9467a6ed89343f1845ff6a7581e6090" member_type: VOTER } }
I20250626 01:51:41.208037 2076 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:41.208040 2077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [sys.catalog]: This master's current role is: LEADER
I20250626 01:51:41.209389 2080 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:51:41.210837 2080 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:51:41.212540 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:51:41.214061 2080 catalog_manager.cc:1349] Generated new cluster ID: bdeb9a4dedc54fc7a33ccb516d5acf35
I20250626 01:51:41.214143 2080 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:51:41.224555 2080 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:51:41.225374 2080 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:51:41.233500 2080 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090: Generated new TSK 0
I20250626 01:51:41.233752 2080 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:51:41.245640 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:41.248737 2093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.249099 2094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.249295 2096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:41.249465 413 server_base.cc:1048] running on GCE node
I20250626 01:51:41.249903 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:41.250001 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:41.250070 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902701250049 us; error 0 us; skew 500 ppm
I20250626 01:51:41.250268 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:41.251691 413 webserver.cc:469] Webserver started at http://127.0.103.65:42105/ using document root <none> and password file <none>
I20250626 01:51:41.252009 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:41.252108 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:41.252290 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:41.252987 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "b2fd87da47df40f78baacd79f25c1a56"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.255652 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:51:41.257527 2101 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.257897 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:51:41.258041 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "b2fd87da47df40f78baacd79f25c1a56"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.258169 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:41.271430 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:41.272114 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:41.272723 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:41.273622 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:41.273730 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.273849 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:41.273929 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.294212 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:37653
I20250626 01:51:41.294234 2163 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:37653 every 8 connection(s)
I20250626 01:51:41.295853 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:41.298738 2169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.298738 2168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.300117 2171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:41.300942 413 server_base.cc:1048] running on GCE node
I20250626 01:51:41.301292 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:41.301369 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:41.301402 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902701301399 us; error 0 us; skew 500 ppm
I20250626 01:51:41.301620 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:41.303031 413 webserver.cc:469] Webserver started at http://127.0.103.66:42455/ using document root <none> and password file <none>
I20250626 01:51:41.303295 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:41.303368 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:41.303488 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:41.303813 2164 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34087
I20250626 01:51:41.303961 2164 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:41.304279 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "a45e2df6f0024748a9eaaa0c9afd2d11"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.304384 2164 heartbeater.cc:507] Master 127.0.103.126:34087 requested a full tablet report, sending...
I20250626 01:51:41.305327 2037 ts_manager.cc:194] Registered new tserver with Master: b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65:37653)
I20250626 01:51:41.306339 2037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48586
I20250626 01:51:41.306972 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:51:41.308599 2176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.308910 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:51:41.309005 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "a45e2df6f0024748a9eaaa0c9afd2d11"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.309123 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:41.320744 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:41.321300 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:41.321830 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:41.322638 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:41.322696 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.322784 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:41.322849 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.335506 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36247
I20250626 01:51:41.335541 2238 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36247 every 8 connection(s)
I20250626 01:51:41.336989 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:41.339826 2243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.340662 2242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:41.343115 2245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:41.344491 2239 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34087
I20250626 01:51:41.344619 2239 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:41.344704 413 server_base.cc:1048] running on GCE node
I20250626 01:51:41.344998 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250626 01:51:41.345077 2239 heartbeater.cc:507] Master 127.0.103.126:34087 requested a full tablet report, sending...
W20250626 01:51:41.345077 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:41.345182 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902701345178 us; error 0 us; skew 500 ppm
I20250626 01:51:41.345326 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:41.346388 2037 ts_manager.cc:194] Registered new tserver with Master: a45e2df6f0024748a9eaaa0c9afd2d11 (127.0.103.66:36247)
I20250626 01:51:41.346834 413 webserver.cc:469] Webserver started at http://127.0.103.67:44353/ using document root <none> and password file <none>
I20250626 01:51:41.347033 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:41.347097 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:41.347210 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:41.347323 2037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48602
I20250626 01:51:41.347980 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "7cd06b2db43645e7baee6a1a4378906b"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.350510 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 01:51:41.351998 2250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.352360 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:51:41.352447 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "7cd06b2db43645e7baee6a1a4378906b"
format_stamp: "Formatted at 2025-06-26 01:51:41 on dist-test-slave-psm5"
I20250626 01:51:41.352557 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:41.363039 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:41.363649 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:41.364284 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:41.365085 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:41.365141 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.365216 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:41.365254 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:41.382185 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:45983
I20250626 01:51:41.382212 2312 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:45983 every 8 connection(s)
I20250626 01:51:41.389289 2313 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34087
I20250626 01:51:41.389441 2313 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:41.389770 2313 heartbeater.cc:507] Master 127.0.103.126:34087 requested a full tablet report, sending...
I20250626 01:51:41.390605 2037 ts_manager.cc:194] Registered new tserver with Master: 7cd06b2db43645e7baee6a1a4378906b (127.0.103.67:45983)
I20250626 01:51:41.391012 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007922032s
I20250626 01:51:41.391417 2037 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48614
I20250626 01:51:42.307673 2164 heartbeater.cc:499] Master 127.0.103.126:34087 was elected leader, sending a full tablet report...
I20250626 01:51:42.348812 2239 heartbeater.cc:499] Master 127.0.103.126:34087 was elected leader, sending a full tablet report...
I20250626 01:51:42.392854 2313 heartbeater.cc:499] Master 127.0.103.126:34087 was elected leader, sending a full tablet report...
I20250626 01:51:42.419587 413 test_util.cc:276] Using random seed: -845336998
I20250626 01:51:42.432317 2037 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48620:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:51:42.433089 2037 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:51:42.455785 2203 tablet_service.cc:1468] Processing CreateTablet for tablet f587f360bec3409d84bdfcb7e42f521f (DEFAULT_TABLE table=test-workload [id=0390882e6b76495989ecaeec4d0804a8]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250626 01:51:42.455785 2204 tablet_service.cc:1468] Processing CreateTablet for tablet ef94e562db4342f1b55f3ac62d988e38 (DEFAULT_TABLE table=test-workload [id=0390882e6b76495989ecaeec4d0804a8]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250626 01:51:42.456652 2203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f587f360bec3409d84bdfcb7e42f521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:42.459183 2204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef94e562db4342f1b55f3ac62d988e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:42.459312 2278 tablet_service.cc:1468] Processing CreateTablet for tablet ef94e562db4342f1b55f3ac62d988e38 (DEFAULT_TABLE table=test-workload [id=0390882e6b76495989ecaeec4d0804a8]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250626 01:51:42.459318 2277 tablet_service.cc:1468] Processing CreateTablet for tablet f587f360bec3409d84bdfcb7e42f521f (DEFAULT_TABLE table=test-workload [id=0390882e6b76495989ecaeec4d0804a8]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250626 01:51:42.459940 2278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef94e562db4342f1b55f3ac62d988e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:42.461334 2277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f587f360bec3409d84bdfcb7e42f521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:42.472546 2129 tablet_service.cc:1468] Processing CreateTablet for tablet ef94e562db4342f1b55f3ac62d988e38 (DEFAULT_TABLE table=test-workload [id=0390882e6b76495989ecaeec4d0804a8]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250626 01:51:42.473289 2129 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef94e562db4342f1b55f3ac62d988e38. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:42.475085 2333 tablet_bootstrap.cc:492] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b: Bootstrap starting.
I20250626 01:51:42.475164 2128 tablet_service.cc:1468] Processing CreateTablet for tablet f587f360bec3409d84bdfcb7e42f521f (DEFAULT_TABLE table=test-workload [id=0390882e6b76495989ecaeec4d0804a8]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250626 01:51:42.475663 2128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f587f360bec3409d84bdfcb7e42f521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:42.477129 2333 tablet_bootstrap.cc:654] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:42.481165 2334 tablet_bootstrap.cc:492] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11: Bootstrap starting.
I20250626 01:51:42.483777 2334 tablet_bootstrap.cc:654] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:42.485723 2334 tablet_bootstrap.cc:492] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11: No bootstrap required, opened a new log
I20250626 01:51:42.485723 2333 tablet_bootstrap.cc:492] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b: No bootstrap required, opened a new log
I20250626 01:51:42.485918 2333 ts_tablet_manager.cc:1397] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20250626 01:51:42.485918 2334 ts_tablet_manager.cc:1397] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250626 01:51:42.486791 2334 raft_consensus.cc:357] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.486791 2333 raft_consensus.cc:357] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.487021 2333 raft_consensus.cc:383] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:42.487022 2334 raft_consensus.cc:383] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:42.487143 2333 raft_consensus.cc:738] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd06b2db43645e7baee6a1a4378906b, State: Initialized, Role: FOLLOWER
I20250626 01:51:42.487198 2334 raft_consensus.cc:738] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45e2df6f0024748a9eaaa0c9afd2d11, State: Initialized, Role: FOLLOWER
I20250626 01:51:42.487458 2334 consensus_queue.cc:260] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.487452 2333 consensus_queue.cc:260] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.489209 2333 ts_tablet_manager.cc:1428] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250626 01:51:42.489765 2336 tablet_bootstrap.cc:492] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56: Bootstrap starting.
I20250626 01:51:42.489871 2334 ts_tablet_manager.cc:1428] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20250626 01:51:42.492198 2334 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: Bootstrap starting.
I20250626 01:51:42.490033 2333 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b: Bootstrap starting.
I20250626 01:51:42.494065 2333 tablet_bootstrap.cc:654] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:42.491853 2336 tablet_bootstrap.cc:654] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:42.494065 2334 tablet_bootstrap.cc:654] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:42.496687 2336 tablet_bootstrap.cc:492] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56: No bootstrap required, opened a new log
I20250626 01:51:42.496722 2333 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b: No bootstrap required, opened a new log
I20250626 01:51:42.496856 2336 ts_tablet_manager.cc:1397] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250626 01:51:42.496856 2333 ts_tablet_manager.cc:1397] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250626 01:51:42.496940 2334 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: No bootstrap required, opened a new log
I20250626 01:51:42.497117 2334 ts_tablet_manager.cc:1397] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250626 01:51:42.497854 2334 raft_consensus.cc:357] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.497809 2336 raft_consensus.cc:357] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.497805 2333 raft_consensus.cc:357] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.498015 2334 raft_consensus.cc:383] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:42.498124 2334 raft_consensus.cc:738] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a45e2df6f0024748a9eaaa0c9afd2d11, State: Initialized, Role: FOLLOWER
I20250626 01:51:42.498021 2336 raft_consensus.cc:383] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:42.498026 2333 raft_consensus.cc:383] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:42.498299 2336 raft_consensus.cc:738] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fd87da47df40f78baacd79f25c1a56, State: Initialized, Role: FOLLOWER
I20250626 01:51:42.498363 2333 raft_consensus.cc:738] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cd06b2db43645e7baee6a1a4378906b, State: Initialized, Role: FOLLOWER
I20250626 01:51:42.498387 2334 consensus_queue.cc:260] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.498550 2336 consensus_queue.cc:260] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.498639 2333 consensus_queue.cc:260] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.498870 2334 ts_tablet_manager.cc:1428] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:51:42.499835 2333 ts_tablet_manager.cc:1428] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250626 01:51:42.504416 2336 ts_tablet_manager.cc:1428] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56: Time spent starting tablet: real 0.007s user 0.000s sys 0.008s
I20250626 01:51:42.504695 2336 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: Bootstrap starting.
I20250626 01:51:42.506475 2336 tablet_bootstrap.cc:654] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: Neither blocks nor log segments found. Creating new log.
I20250626 01:51:42.508548 2336 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: No bootstrap required, opened a new log
I20250626 01:51:42.508692 2336 ts_tablet_manager.cc:1397] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250626 01:51:42.509307 2336 raft_consensus.cc:357] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.509452 2336 raft_consensus.cc:383] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:51:42.509521 2336 raft_consensus.cc:738] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2fd87da47df40f78baacd79f25c1a56, State: Initialized, Role: FOLLOWER
I20250626 01:51:42.509728 2336 consensus_queue.cc:260] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.510140 2336 ts_tablet_manager.cc:1428] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:51:42.511713 2343 raft_consensus.cc:491] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:42.511831 2343 raft_consensus.cc:513] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.512630 2343 leader_election.cc:290] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a45e2df6f0024748a9eaaa0c9afd2d11 (127.0.103.66:36247), 7cd06b2db43645e7baee6a1a4378906b (127.0.103.67:45983)
I20250626 01:51:42.520294 2288 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef94e562db4342f1b55f3ac62d988e38" candidate_uuid: "b2fd87da47df40f78baacd79f25c1a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd06b2db43645e7baee6a1a4378906b" is_pre_election: true
I20250626 01:51:42.520516 2288 raft_consensus.cc:2466] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2fd87da47df40f78baacd79f25c1a56 in term 0.
I20250626 01:51:42.520818 2214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef94e562db4342f1b55f3ac62d988e38" candidate_uuid: "b2fd87da47df40f78baacd79f25c1a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" is_pre_election: true
I20250626 01:51:42.520924 2104 leader_election.cc:304] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cd06b2db43645e7baee6a1a4378906b, b2fd87da47df40f78baacd79f25c1a56; no voters:
I20250626 01:51:42.520989 2214 raft_consensus.cc:2466] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2fd87da47df40f78baacd79f25c1a56 in term 0.
I20250626 01:51:42.521217 2343 raft_consensus.cc:2802] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:42.521307 2343 raft_consensus.cc:491] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:42.521382 2343 raft_consensus.cc:3058] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:42.523490 2343 raft_consensus.cc:513] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.524035 2343 leader_election.cc:290] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [CANDIDATE]: Term 1 election: Requested vote from peers a45e2df6f0024748a9eaaa0c9afd2d11 (127.0.103.66:36247), 7cd06b2db43645e7baee6a1a4378906b (127.0.103.67:45983)
I20250626 01:51:42.524413 2214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef94e562db4342f1b55f3ac62d988e38" candidate_uuid: "b2fd87da47df40f78baacd79f25c1a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11"
I20250626 01:51:42.524466 2288 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ef94e562db4342f1b55f3ac62d988e38" candidate_uuid: "b2fd87da47df40f78baacd79f25c1a56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd06b2db43645e7baee6a1a4378906b"
I20250626 01:51:42.524541 2214 raft_consensus.cc:3058] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:42.524657 2288 raft_consensus.cc:3058] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:42.526515 2214 raft_consensus.cc:2466] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fd87da47df40f78baacd79f25c1a56 in term 1.
I20250626 01:51:42.526515 2288 raft_consensus.cc:2466] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2fd87da47df40f78baacd79f25c1a56 in term 1.
I20250626 01:51:42.526906 2104 leader_election.cc:304] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a45e2df6f0024748a9eaaa0c9afd2d11, b2fd87da47df40f78baacd79f25c1a56; no voters:
I20250626 01:51:42.527231 2343 raft_consensus.cc:2802] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:42.527634 2343 raft_consensus.cc:695] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 1 LEADER]: Becoming Leader. State: Replica: b2fd87da47df40f78baacd79f25c1a56, State: Running, Role: LEADER
I20250626 01:51:42.527936 2343 consensus_queue.cc:237] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.531210 2037 catalog_manager.cc:5582] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "b2fd87da47df40f78baacd79f25c1a56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:42.633033 2339 raft_consensus.cc:491] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:42.633308 2339 raft_consensus.cc:513] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.634294 2339 leader_election.cc:290] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65:37653), 7cd06b2db43645e7baee6a1a4378906b (127.0.103.67:45983)
I20250626 01:51:42.634336 2338 raft_consensus.cc:491] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:51:42.634514 2338 raft_consensus.cc:513] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.635528 2338 leader_election.cc:290] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65:37653), a45e2df6f0024748a9eaaa0c9afd2d11 (127.0.103.66:36247)
I20250626 01:51:42.648794 2139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f587f360bec3409d84bdfcb7e42f521f" candidate_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2fd87da47df40f78baacd79f25c1a56" is_pre_election: true
I20250626 01:51:42.648891 2288 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f587f360bec3409d84bdfcb7e42f521f" candidate_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd06b2db43645e7baee6a1a4378906b" is_pre_election: true
I20250626 01:51:42.649225 2288 raft_consensus.cc:2466] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a45e2df6f0024748a9eaaa0c9afd2d11 in term 0.
I20250626 01:51:42.649222 2139 raft_consensus.cc:2466] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a45e2df6f0024748a9eaaa0c9afd2d11 in term 0.
I20250626 01:51:42.650745 2179 leader_election.cc:304] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7cd06b2db43645e7baee6a1a4378906b, a45e2df6f0024748a9eaaa0c9afd2d11, b2fd87da47df40f78baacd79f25c1a56; no voters:
I20250626 01:51:42.651152 2339 raft_consensus.cc:2802] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:51:42.651283 2339 raft_consensus.cc:491] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:51:42.651412 2339 raft_consensus.cc:3058] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:42.654449 2339 raft_consensus.cc:513] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.655576 2339 leader_election.cc:290] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [CANDIDATE]: Term 1 election: Requested vote from peers b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65:37653), 7cd06b2db43645e7baee6a1a4378906b (127.0.103.67:45983)
I20250626 01:51:42.656040 2139 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f587f360bec3409d84bdfcb7e42f521f" candidate_uuid: "7cd06b2db43645e7baee6a1a4378906b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2fd87da47df40f78baacd79f25c1a56" is_pre_election: true
I20250626 01:51:42.656097 2138 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f587f360bec3409d84bdfcb7e42f521f" candidate_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2fd87da47df40f78baacd79f25c1a56"
I20250626 01:51:42.656316 2288 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f587f360bec3409d84bdfcb7e42f521f" candidate_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cd06b2db43645e7baee6a1a4378906b"
I20250626 01:51:42.656487 2214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f587f360bec3409d84bdfcb7e42f521f" candidate_uuid: "7cd06b2db43645e7baee6a1a4378906b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" is_pre_election: true
I20250626 01:51:42.656550 2288 raft_consensus.cc:3058] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:42.656747 2214 raft_consensus.cc:2391] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7cd06b2db43645e7baee6a1a4378906b in current term 1: Already voted for candidate a45e2df6f0024748a9eaaa0c9afd2d11 in this term.
I20250626 01:51:42.656318 2139 raft_consensus.cc:2466] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cd06b2db43645e7baee6a1a4378906b in term 0.
I20250626 01:51:42.657436 2254 leader_election.cc:304] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7cd06b2db43645e7baee6a1a4378906b, b2fd87da47df40f78baacd79f25c1a56; no voters: a45e2df6f0024748a9eaaa0c9afd2d11
I20250626 01:51:42.658974 2288 raft_consensus.cc:2466] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a45e2df6f0024748a9eaaa0c9afd2d11 in term 1.
I20250626 01:51:42.659148 2338 raft_consensus.cc:2762] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250626 01:51:42.659430 2179 leader_election.cc:304] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7cd06b2db43645e7baee6a1a4378906b, a45e2df6f0024748a9eaaa0c9afd2d11; no voters: b2fd87da47df40f78baacd79f25c1a56
I20250626 01:51:42.659745 2339 raft_consensus.cc:2802] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:51:42.660162 2339 raft_consensus.cc:695] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 LEADER]: Becoming Leader. State: Replica: a45e2df6f0024748a9eaaa0c9afd2d11, State: Running, Role: LEADER
I20250626 01:51:42.660666 2339 consensus_queue.cc:237] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } }
I20250626 01:51:42.663204 2037 catalog_manager.cc:5582] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 reported cstate change: term changed from 0 to 1, leader changed from <none> to a45e2df6f0024748a9eaaa0c9afd2d11 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } health_report { overall_health: UNKNOWN } } }
I20250626 01:51:43.097571 2356 consensus_queue.cc:1035] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:43.102860 2360 consensus_queue.cc:1035] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:43.146241 2339 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:43.149773 2139 raft_consensus.cc:3058] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:51:43.153370 2365 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:51:43.212373 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:43.212960 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:43.213117 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:45.213660 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:45.214399 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:45.214565 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:47.215061 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:47.215713 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:47.215857 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:49.216382 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:49.217039 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:49.217171 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:51.217705 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:51.218437 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:51.218604 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:53.219108 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:53.219830 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:53.219980 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:53.738145 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:51:53.742811 2374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:53.742839 2375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:51:53.743250 2377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:51:53.745621 413 server_base.cc:1048] running on GCE node
I20250626 01:51:53.746488 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:51:53.746564 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:51:53.746613 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902713746598 us; error 0 us; skew 500 ppm
I20250626 01:51:53.746757 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:51:53.748261 413 webserver.cc:469] Webserver started at http://127.0.103.68:42447/ using document root <none> and password file <none>
I20250626 01:51:53.748507 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:51:53.748582 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:51:53.748710 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:51:53.749336 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "22faf1c0daad4b969a26fc17da4a9ad4"
format_stamp: "Formatted at 2025-06-26 01:51:53 on dist-test-slave-psm5"
I20250626 01:51:53.752070 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:51:53.753772 2382 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:53.754199 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:51:53.754330 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "22faf1c0daad4b969a26fc17da4a9ad4"
format_stamp: "Formatted at 2025-06-26 01:51:53 on dist-test-slave-psm5"
I20250626 01:51:53.754462 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:51:53.772006 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:51:53.772651 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:51:53.773281 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:51:53.774190 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:51:53.774286 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:53.774391 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:51:53.774468 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:51:53.792541 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:34163
I20250626 01:51:53.792558 2444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:34163 every 8 connection(s)
I20250626 01:51:53.800828 2445 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34087
I20250626 01:51:53.801005 2445 heartbeater.cc:461] Registering TS with master...
I20250626 01:51:53.801424 2445 heartbeater.cc:507] Master 127.0.103.126:34087 requested a full tablet report, sending...
I20250626 01:51:53.802512 2036 ts_manager.cc:194] Registered new tserver with Master: 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68:34163)
I20250626 01:51:53.803400 2036 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46372
I20250626 01:51:54.235219 2214 consensus_queue.cc:237] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } }
I20250626 01:51:54.238251 2139 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:54.239115 2453 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:54.239837 2288 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:51:54.240476 2452 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:51:54.242763 2453 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } } }
I20250626 01:51:54.243494 2139 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } } }
I20250626 01:51:54.247099 2288 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } } }
W20250626 01:51:54.249423 2179 consensus_peers.cc:487] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 -> Peer 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68:34163): Couldn't send request to peer 22faf1c0daad4b969a26fc17da4a9ad4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f587f360bec3409d84bdfcb7e42f521f. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:51:54.249336 2036 catalog_manager.cc:5582] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 reported cstate change: config changed from index -1 to 2, NON_VOTER 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } } }
I20250626 01:51:54.804738 2445 heartbeater.cc:499] Master 127.0.103.126:34087 was elected leader, sending a full tablet report...
I20250626 01:51:54.852283 2464 ts_tablet_manager.cc:927] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: Initiating tablet copy from peer a45e2df6f0024748a9eaaa0c9afd2d11 (127.0.103.66:36247)
I20250626 01:51:54.853279 2464 tablet_copy_client.cc:323] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:36247
I20250626 01:51:54.860009 2224 tablet_copy_service.cc:140] P a45e2df6f0024748a9eaaa0c9afd2d11: Received BeginTabletCopySession request for tablet f587f360bec3409d84bdfcb7e42f521f from peer 22faf1c0daad4b969a26fc17da4a9ad4 ({username='slave'} at 127.0.0.1:38970)
I20250626 01:51:54.860245 2224 tablet_copy_service.cc:161] P a45e2df6f0024748a9eaaa0c9afd2d11: Beginning new tablet copy session on tablet f587f360bec3409d84bdfcb7e42f521f from peer 22faf1c0daad4b969a26fc17da4a9ad4 at {username='slave'} at 127.0.0.1:38970: session id = 22faf1c0daad4b969a26fc17da4a9ad4-f587f360bec3409d84bdfcb7e42f521f
I20250626 01:51:54.863242 2224 tablet_copy_source_session.cc:215] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:51:54.865152 2464 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f587f360bec3409d84bdfcb7e42f521f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:51:54.870718 2464 tablet_copy_client.cc:806] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: tablet copy: Starting download of 0 data blocks...
I20250626 01:51:54.871078 2464 tablet_copy_client.cc:670] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:51:54.873332 2464 tablet_copy_client.cc:538] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:51:54.877252 2464 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: Bootstrap starting.
I20250626 01:51:54.884471 2464 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:51:54.884876 2464 tablet_bootstrap.cc:492] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: Bootstrap complete.
I20250626 01:51:54.885121 2464 ts_tablet_manager.cc:1397] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20250626 01:51:54.885902 2464 raft_consensus.cc:357] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } }
I20250626 01:51:54.886086 2464 raft_consensus.cc:738] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22faf1c0daad4b969a26fc17da4a9ad4, State: Initialized, Role: LEARNER
I20250626 01:51:54.886276 2464 consensus_queue.cc:260] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: true } }
I20250626 01:51:54.887071 2464 ts_tablet_manager.cc:1428] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:51:54.887972 2224 tablet_copy_service.cc:342] P a45e2df6f0024748a9eaaa0c9afd2d11: Request end of tablet copy session 22faf1c0daad4b969a26fc17da4a9ad4-f587f360bec3409d84bdfcb7e42f521f received from {username='slave'} at 127.0.0.1:38970
I20250626 01:51:54.888145 2224 tablet_copy_service.cc:434] P a45e2df6f0024748a9eaaa0c9afd2d11: ending tablet copy session 22faf1c0daad4b969a26fc17da4a9ad4-f587f360bec3409d84bdfcb7e42f521f on tablet f587f360bec3409d84bdfcb7e42f521f with peer 22faf1c0daad4b969a26fc17da4a9ad4
I20250626 01:51:55.220518 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:55.221232 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:55.221391 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 01:51:55.303462 2309 debug-util.cc:398] Leaking SignalData structure 0x603000fdd1a0 after lost signal to thread 2448
I20250626 01:51:55.313349 2420 raft_consensus.cc:1215] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 01:51:55.720633 2463 raft_consensus.cc:1062] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: attempting to promote NON_VOTER 22faf1c0daad4b969a26fc17da4a9ad4 to VOTER
I20250626 01:51:55.721796 2463 consensus_queue.cc:237] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } }
I20250626 01:51:55.724179 2420 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 LEARNER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:51:55.725142 2288 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:51:55.725288 2470 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:55.726024 2470 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:55.725155 2139 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:51:55.729306 2470 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:51:55.742089 2470 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.744470 2420 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.745627 2287 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.752462 2037 catalog_manager.cc:5582] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 reported cstate change: config changed from index 2 to 3, 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.754648 2139 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 22faf1c0daad4b969a26fc17da4a9ad4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b2fd87da47df40f78baacd79f25c1a56" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37653 } attrs { replace: true } } peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.770689 2213 consensus_queue.cc:237] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } }
I20250626 01:51:55.773044 2420 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 FOLLOWER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:51:55.773139 2287 raft_consensus.cc:1273] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Refusing update from remote peer a45e2df6f0024748a9eaaa0c9afd2d11: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:51:55.773672 2470 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:51:55.774536 2477 consensus_queue.cc:1035] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:51:55.776152 2472 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.776690 2287 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.777575 2420 raft_consensus.cc:2953] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.780781 2023 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f587f360bec3409d84bdfcb7e42f521f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:51:55.782407 2035 catalog_manager.cc:5582] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b reported cstate change: config changed from index 3 to 4, VOTER b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a45e2df6f0024748a9eaaa0c9afd2d11" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36247 } } peers { permanent_uuid: "7cd06b2db43645e7baee6a1a4378906b" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 45983 } } peers { permanent_uuid: "22faf1c0daad4b969a26fc17da4a9ad4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34163 } attrs { promote: false } } }
I20250626 01:51:55.786687 2129 tablet_service.cc:1515] Processing DeleteTablet for tablet f587f360bec3409d84bdfcb7e42f521f with delete_type TABLET_DATA_TOMBSTONED (TS b2fd87da47df40f78baacd79f25c1a56 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47020
I20250626 01:51:55.787814 2482 tablet_replica.cc:331] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: stopping tablet replica
I20250626 01:51:55.788064 2482 raft_consensus.cc:2241] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:51:55.788379 2482 raft_consensus.cc:2270] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:51:55.789367 2482 ts_tablet_manager.cc:1905] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:51:55.794777 2482 ts_tablet_manager.cc:1918] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 01:51:55.794961 2482 log.cc:1199] T f587f360bec3409d84bdfcb7e42f521f P b2fd87da47df40f78baacd79f25c1a56: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1750902684575999-413-0/minicluster-data/ts-0-root/wals/f587f360bec3409d84bdfcb7e42f521f
I20250626 01:51:55.796113 2024 catalog_manager.cc:4928] TS b2fd87da47df40f78baacd79f25c1a56 (127.0.103.65:37653): tablet f587f360bec3409d84bdfcb7e42f521f (table test-workload [id=0390882e6b76495989ecaeec4d0804a8]) successfully deleted
I20250626 01:51:57.221793 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:57.222466 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:57.222615 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:51:59.223150 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:51:59.223801 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:51:59.224020 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:01.224530 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:01.225212 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:01.225359 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:03.225934 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:03.226562 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:03.226706 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:05.227213 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:05.227906 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:05.228034 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:07.228547 2090 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:07.229233 2090 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:07.229374 2090 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:07.411237 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:52:07.428877 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:07.429241 413 tablet_replica.cc:331] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56: stopping tablet replica
I20250626 01:52:07.429392 413 raft_consensus.cc:2241] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:07.429708 413 raft_consensus.cc:2270] T ef94e562db4342f1b55f3ac62d988e38 P b2fd87da47df40f78baacd79f25c1a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.447913 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:52:07.451426 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:52:07.463454 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:07.463833 413 tablet_replica.cc:331] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11: stopping tablet replica
I20250626 01:52:07.464043 413 raft_consensus.cc:2241] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:07.464380 413 raft_consensus.cc:2270] T f587f360bec3409d84bdfcb7e42f521f P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.464972 413 tablet_replica.cc:331] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11: stopping tablet replica
I20250626 01:52:07.465107 413 raft_consensus.cc:2241] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:07.465253 413 raft_consensus.cc:2270] T ef94e562db4342f1b55f3ac62d988e38 P a45e2df6f0024748a9eaaa0c9afd2d11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.480389 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:52:07.483734 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:52:07.493589 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:07.493875 413 tablet_replica.cc:331] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b: stopping tablet replica
I20250626 01:52:07.494046 413 raft_consensus.cc:2241] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:07.494314 413 raft_consensus.cc:2270] T f587f360bec3409d84bdfcb7e42f521f P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.494899 413 tablet_replica.cc:331] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b: stopping tablet replica
I20250626 01:52:07.495010 413 raft_consensus.cc:2241] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:07.495167 413 raft_consensus.cc:2270] T ef94e562db4342f1b55f3ac62d988e38 P 7cd06b2db43645e7baee6a1a4378906b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.509869 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:52:07.513316 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:52:07.523100 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:07.523406 413 tablet_replica.cc:331] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4: stopping tablet replica
I20250626 01:52:07.523576 413 raft_consensus.cc:2241] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:07.523823 413 raft_consensus.cc:2270] T f587f360bec3409d84bdfcb7e42f521f P 22faf1c0daad4b969a26fc17da4a9ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.537381 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:52:07.540536 413 master.cc:561] Master@127.0.103.126:34087 shutting down...
I20250626 01:52:07.549041 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:07.549347 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:07.549443 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P c9467a6ed89343f1845ff6a7581e6090: stopping tablet replica
I20250626 01:52:07.563269 413 master.cc:583] Master@127.0.103.126:34087 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26431 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250626 01:52:07.580721 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46391
I20250626 01:52:07.581283 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:07.583842 2487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:07.583858 2488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:07.584172 413 server_base.cc:1048] running on GCE node
W20250626 01:52:07.584129 2490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:07.584746 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:07.584813 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:07.584838 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902727584835 us; error 0 us; skew 500 ppm
I20250626 01:52:07.584975 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:07.589625 413 webserver.cc:469] Webserver started at http://127.0.103.126:46031/ using document root <none> and password file <none>
I20250626 01:52:07.589869 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:07.589954 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:07.590061 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:07.590674 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "5d554bd5d91947089def31af7956c4dd"
format_stamp: "Formatted at 2025-06-26 01:52:07 on dist-test-slave-psm5"
I20250626 01:52:07.593385 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:52:07.594975 2495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:07.595341 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:52:07.595443 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "5d554bd5d91947089def31af7956c4dd"
format_stamp: "Formatted at 2025-06-26 01:52:07 on dist-test-slave-psm5"
I20250626 01:52:07.595558 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:07.625705 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:07.626329 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:07.638849 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46391
I20250626 01:52:07.638906 2546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:46391 every 8 connection(s)
I20250626 01:52:07.640286 2547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:07.644764 2547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd: Bootstrap starting.
I20250626 01:52:07.646008 2547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:07.647648 2547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd: No bootstrap required, opened a new log
I20250626 01:52:07.648412 2547 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d554bd5d91947089def31af7956c4dd" member_type: VOTER }
I20250626 01:52:07.648558 2547 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:07.648605 2547 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d554bd5d91947089def31af7956c4dd, State: Initialized, Role: FOLLOWER
I20250626 01:52:07.648799 2547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d554bd5d91947089def31af7956c4dd" member_type: VOTER }
I20250626 01:52:07.648927 2547 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:52:07.648977 2547 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:52:07.649037 2547 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:07.650825 2547 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d554bd5d91947089def31af7956c4dd" member_type: VOTER }
I20250626 01:52:07.651023 2547 leader_election.cc:304] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d554bd5d91947089def31af7956c4dd; no voters:
I20250626 01:52:07.651423 2547 leader_election.cc:290] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:52:07.651602 2550 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:07.652070 2550 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 1 LEADER]: Becoming Leader. State: Replica: 5d554bd5d91947089def31af7956c4dd, State: Running, Role: LEADER
I20250626 01:52:07.652349 2550 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d554bd5d91947089def31af7956c4dd" member_type: VOTER }
I20250626 01:52:07.652446 2547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:52:07.653512 2551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d554bd5d91947089def31af7956c4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d554bd5d91947089def31af7956c4dd" member_type: VOTER } }
I20250626 01:52:07.653573 2552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d554bd5d91947089def31af7956c4dd. Latest consensus state: current_term: 1 leader_uuid: "5d554bd5d91947089def31af7956c4dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d554bd5d91947089def31af7956c4dd" member_type: VOTER } }
I20250626 01:52:07.653676 2551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:07.653683 2552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:07.654747 2555 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:52:07.656857 2555 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:52:07.658241 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:52:07.660904 2555 catalog_manager.cc:1349] Generated new cluster ID: 95ff44478da047d5a3f29c1250b9edcf
I20250626 01:52:07.661016 2555 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:52:07.677635 2555 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:52:07.678483 2555 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:52:07.687147 2555 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd: Generated new TSK 0
I20250626 01:52:07.687435 2555 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:52:07.690789 413 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 8.548e-05s
I20250626 01:52:19.774446 413 master.cc:561] Master@127.0.103.126:46391 shutting down...
I20250626 01:52:19.788861 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:19.789109 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:19.789194 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5d554bd5d91947089def31af7956c4dd: stopping tablet replica
I20250626 01:52:19.803874 413 master.cc:583] Master@127.0.103.126:46391 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12231 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250626 01:52:19.812080 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:40287
I20250626 01:52:19.812647 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:19.815501 2570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:19.815474 2569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:19.815479 2572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:19.819303 413 server_base.cc:1048] running on GCE node
I20250626 01:52:19.821492 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:19.821586 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:19.821625 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902739821610 us; error 0 us; skew 500 ppm
I20250626 01:52:19.821768 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:19.823166 413 webserver.cc:469] Webserver started at http://127.0.103.126:40335/ using document root <none> and password file <none>
I20250626 01:52:19.823407 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:19.823479 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:19.823589 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:19.824354 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "6adf99faeff6452bb9ee67fe17d7900b"
format_stamp: "Formatted at 2025-06-26 01:52:19 on dist-test-slave-psm5"
I20250626 01:52:19.827059 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:52:19.828814 2577 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:19.829277 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:52:19.829416 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "6adf99faeff6452bb9ee67fe17d7900b"
format_stamp: "Formatted at 2025-06-26 01:52:19 on dist-test-slave-psm5"
I20250626 01:52:19.829545 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:19.842962 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:19.843549 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:19.857123 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:40287
I20250626 01:52:19.857156 2628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:40287 every 8 connection(s)
I20250626 01:52:19.858569 2629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:19.862847 2629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b: Bootstrap starting.
I20250626 01:52:19.864130 2629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:19.865847 2629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b: No bootstrap required, opened a new log
I20250626 01:52:19.866525 2629 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adf99faeff6452bb9ee67fe17d7900b" member_type: VOTER }
I20250626 01:52:19.866673 2629 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:19.866763 2629 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adf99faeff6452bb9ee67fe17d7900b, State: Initialized, Role: FOLLOWER
I20250626 01:52:19.866991 2629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adf99faeff6452bb9ee67fe17d7900b" member_type: VOTER }
I20250626 01:52:19.867139 2629 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:52:19.867226 2629 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:52:19.867329 2629 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:19.869221 2629 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adf99faeff6452bb9ee67fe17d7900b" member_type: VOTER }
I20250626 01:52:19.869439 2629 leader_election.cc:304] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6adf99faeff6452bb9ee67fe17d7900b; no voters:
I20250626 01:52:19.869903 2629 leader_election.cc:290] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:52:19.870074 2632 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:19.870558 2632 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 1 LEADER]: Becoming Leader. State: Replica: 6adf99faeff6452bb9ee67fe17d7900b, State: Running, Role: LEADER
I20250626 01:52:19.870812 2629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:52:19.870893 2632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adf99faeff6452bb9ee67fe17d7900b" member_type: VOTER }
I20250626 01:52:19.872385 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6adf99faeff6452bb9ee67fe17d7900b. Latest consensus state: current_term: 1 leader_uuid: "6adf99faeff6452bb9ee67fe17d7900b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adf99faeff6452bb9ee67fe17d7900b" member_type: VOTER } }
I20250626 01:52:19.872323 2633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6adf99faeff6452bb9ee67fe17d7900b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6adf99faeff6452bb9ee67fe17d7900b" member_type: VOTER } }
I20250626 01:52:19.872563 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:19.872592 2633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:19.873689 2637 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:52:19.875064 2637 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:52:19.877511 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:52:19.878460 2637 catalog_manager.cc:1349] Generated new cluster ID: 009d871bce13405a8f65f08860bea97d
I20250626 01:52:19.878569 2637 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:52:19.889056 2637 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:52:19.889995 2637 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:52:19.902644 2637 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b: Generated new TSK 0
I20250626 01:52:19.902985 2637 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:52:19.910835 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:19.913805 2650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:19.914204 2651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:19.914786 2653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:19.915030 413 server_base.cc:1048] running on GCE node
I20250626 01:52:19.915298 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:19.915364 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:19.915427 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902739915407 us; error 0 us; skew 500 ppm
I20250626 01:52:19.915606 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:19.917161 413 webserver.cc:469] Webserver started at http://127.0.103.65:40197/ using document root <none> and password file <none>
I20250626 01:52:19.917451 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:19.917539 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:19.917692 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:19.918354 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "a67cd3f9a25049988784757672d13633"
format_stamp: "Formatted at 2025-06-26 01:52:19 on dist-test-slave-psm5"
I20250626 01:52:19.921166 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250626 01:52:19.922899 2658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:19.923316 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:52:19.923445 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "a67cd3f9a25049988784757672d13633"
format_stamp: "Formatted at 2025-06-26 01:52:19 on dist-test-slave-psm5"
I20250626 01:52:19.923571 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:19.948370 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:19.948954 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:19.949549 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:19.950448 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:19.950531 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:19.950645 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:19.950706 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:19.965595 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39565
I20250626 01:52:19.965628 2720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39565 every 8 connection(s)
I20250626 01:52:19.967350 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:19.971688 2725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:19.973457 2726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:19.974709 2728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:19.975613 413 server_base.cc:1048] running on GCE node
I20250626 01:52:19.975963 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:19.976050 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:19.976100 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902739976076 us; error 0 us; skew 500 ppm
I20250626 01:52:19.976307 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:19.976333 2721 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40287
I20250626 01:52:19.976536 2721 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:19.976943 2721 heartbeater.cc:507] Master 127.0.103.126:40287 requested a full tablet report, sending...
I20250626 01:52:19.978017 2594 ts_manager.cc:194] Registered new tserver with Master: a67cd3f9a25049988784757672d13633 (127.0.103.65:39565)
I20250626 01:52:19.979135 2594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33640
I20250626 01:52:19.979489 413 webserver.cc:469] Webserver started at http://127.0.103.66:38247/ using document root <none> and password file <none>
I20250626 01:52:19.979761 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:19.979936 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:19.980108 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:19.980770 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "fedcdbbd294d4b90a439a2ab6e437d3b"
format_stamp: "Formatted at 2025-06-26 01:52:19 on dist-test-slave-psm5"
I20250626 01:52:19.983448 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20250626 01:52:19.985257 2733 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:19.985630 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:52:19.985759 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "fedcdbbd294d4b90a439a2ab6e437d3b"
format_stamp: "Formatted at 2025-06-26 01:52:19 on dist-test-slave-psm5"
I20250626 01:52:19.985884 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:19.998739 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:19.999444 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:20.000118 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:20.001147 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:20.001242 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:20.001382 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:20.001448 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:20.015703 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43559
I20250626 01:52:20.015717 2795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:43559 every 8 connection(s)
I20250626 01:52:20.017608 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:20.021066 2799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:20.021144 2800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:20.023118 2802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:20.023522 413 server_base.cc:1048] running on GCE node
I20250626 01:52:20.023837 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:20.023968 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:20.024039 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902740024020 us; error 0 us; skew 500 ppm
I20250626 01:52:20.024247 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:20.025882 2796 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40287
I20250626 01:52:20.026052 2796 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:20.026437 2796 heartbeater.cc:507] Master 127.0.103.126:40287 requested a full tablet report, sending...
I20250626 01:52:20.026976 413 webserver.cc:469] Webserver started at http://127.0.103.67:42387/ using document root <none> and password file <none>
I20250626 01:52:20.027294 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:20.027408 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:20.027558 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:20.027549 2594 ts_manager.cc:194] Registered new tserver with Master: fedcdbbd294d4b90a439a2ab6e437d3b (127.0.103.66:43559)
I20250626 01:52:20.028414 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "d308f35aeee448f780a6da18eefc81d3"
format_stamp: "Formatted at 2025-06-26 01:52:20 on dist-test-slave-psm5"
I20250626 01:52:20.028720 2594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33654
I20250626 01:52:20.031539 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.002s
I20250626 01:52:20.033293 2807 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:20.033665 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:52:20.033782 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "d308f35aeee448f780a6da18eefc81d3"
format_stamp: "Formatted at 2025-06-26 01:52:20 on dist-test-slave-psm5"
I20250626 01:52:20.033928 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:20.048863 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:20.049489 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:20.050053 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:20.051066 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:20.051155 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:20.051281 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:20.051344 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:20.064471 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:46035
I20250626 01:52:20.064486 2869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:46035 every 8 connection(s)
I20250626 01:52:20.071476 2870 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40287
I20250626 01:52:20.071678 2870 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:20.072160 2870 heartbeater.cc:507] Master 127.0.103.126:40287 requested a full tablet report, sending...
I20250626 01:52:20.073058 2594 ts_manager.cc:194] Registered new tserver with Master: d308f35aeee448f780a6da18eefc81d3 (127.0.103.67:46035)
I20250626 01:52:20.073446 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00794013s
I20250626 01:52:20.074074 2594 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33664
I20250626 01:52:20.980824 2721 heartbeater.cc:499] Master 127.0.103.126:40287 was elected leader, sending a full tablet report...
I20250626 01:52:21.030385 2796 heartbeater.cc:499] Master 127.0.103.126:40287 was elected leader, sending a full tablet report...
I20250626 01:52:21.075596 2870 heartbeater.cc:499] Master 127.0.103.126:40287 was elected leader, sending a full tablet report...
I20250626 01:52:21.877197 2647 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:23.877535 2647 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:25.878136 2647 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:27.878701 2647 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:29.879230 2647 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:31.879743 2647 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:32.157930 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:52:32.166863 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:32.179958 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:52:32.182799 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:52:32.190467 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:32.203409 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:52:32.206137 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:52:32.214498 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:32.227401 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:52:32.230257 413 master.cc:561] Master@127.0.103.126:40287 shutting down...
I20250626 01:52:32.237561 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:32.237808 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:32.237912 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 6adf99faeff6452bb9ee67fe17d7900b: stopping tablet replica
I20250626 01:52:32.251452 413 master.cc:583] Master@127.0.103.126:40287 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12448 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250626 01:52:32.261126 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:43443
I20250626 01:52:32.261684 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:32.264233 2874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.264236 2875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.264714 2877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:32.264961 413 server_base.cc:1048] running on GCE node
I20250626 01:52:32.265249 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:32.265300 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:32.265327 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902752265324 us; error 0 us; skew 500 ppm
I20250626 01:52:32.265468 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:32.266815 413 webserver.cc:469] Webserver started at http://127.0.103.126:46317/ using document root <none> and password file <none>
I20250626 01:52:32.267048 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:32.267112 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:32.267258 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:32.267941 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "c769ab463fb44d91b64ffe994dea3291"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.270493 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:32.272208 2882 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.272562 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:52:32.272666 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "c769ab463fb44d91b64ffe994dea3291"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.272779 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:32.295991 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:32.296625 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:32.308701 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:43443
I20250626 01:52:32.308722 2933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:43443 every 8 connection(s)
I20250626 01:52:32.310019 2934 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:32.314047 2934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291: Bootstrap starting.
I20250626 01:52:32.315431 2934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:32.317152 2934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291: No bootstrap required, opened a new log
I20250626 01:52:32.317799 2934 raft_consensus.cc:357] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c769ab463fb44d91b64ffe994dea3291" member_type: VOTER }
I20250626 01:52:32.317940 2934 raft_consensus.cc:383] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:32.317996 2934 raft_consensus.cc:738] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c769ab463fb44d91b64ffe994dea3291, State: Initialized, Role: FOLLOWER
I20250626 01:52:32.318199 2934 consensus_queue.cc:260] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c769ab463fb44d91b64ffe994dea3291" member_type: VOTER }
I20250626 01:52:32.318310 2934 raft_consensus.cc:397] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:52:32.318370 2934 raft_consensus.cc:491] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:52:32.318434 2934 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:32.320211 2934 raft_consensus.cc:513] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c769ab463fb44d91b64ffe994dea3291" member_type: VOTER }
I20250626 01:52:32.320394 2934 leader_election.cc:304] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c769ab463fb44d91b64ffe994dea3291; no voters:
I20250626 01:52:32.320799 2934 leader_election.cc:290] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:52:32.320948 2937 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:32.321424 2937 raft_consensus.cc:695] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 1 LEADER]: Becoming Leader. State: Replica: c769ab463fb44d91b64ffe994dea3291, State: Running, Role: LEADER
I20250626 01:52:32.321657 2934 sys_catalog.cc:564] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:52:32.321631 2937 consensus_queue.cc:237] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c769ab463fb44d91b64ffe994dea3291" member_type: VOTER }
I20250626 01:52:32.322540 2938 sys_catalog.cc:455] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c769ab463fb44d91b64ffe994dea3291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c769ab463fb44d91b64ffe994dea3291" member_type: VOTER } }
I20250626 01:52:32.322590 2939 sys_catalog.cc:455] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c769ab463fb44d91b64ffe994dea3291. Latest consensus state: current_term: 1 leader_uuid: "c769ab463fb44d91b64ffe994dea3291" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c769ab463fb44d91b64ffe994dea3291" member_type: VOTER } }
I20250626 01:52:32.322717 2938 sys_catalog.cc:458] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:32.322734 2939 sys_catalog.cc:458] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:32.328481 2942 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:52:32.329963 2942 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:52:32.332022 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:52:32.333335 2942 catalog_manager.cc:1349] Generated new cluster ID: 1b5fdf083643463a96d8665b11a53889
I20250626 01:52:32.333426 2942 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:52:32.364038 2942 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:52:32.364840 2942 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:52:32.378952 2942 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291: Generated new TSK 0
I20250626 01:52:32.379215 2942 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:52:32.397205 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:32.400168 2955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.400363 2956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:32.401438 413 server_base.cc:1048] running on GCE node
W20250626 01:52:32.401494 2958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:32.401845 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:32.401906 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:32.401942 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902752401938 us; error 0 us; skew 500 ppm
I20250626 01:52:32.402104 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:32.403600 413 webserver.cc:469] Webserver started at http://127.0.103.65:34895/ using document root <none> and password file <none>
I20250626 01:52:32.403836 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:32.403934 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:32.404078 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:32.404718 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.407439 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:52:32.409183 2963 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.409545 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:52:32.409641 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.409761 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:32.425699 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:32.426362 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:32.426968 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:32.427838 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:32.427961 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.428057 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:32.428118 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.442202 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:36413
I20250626 01:52:32.442205 3025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:36413 every 8 connection(s)
I20250626 01:52:32.443984 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:32.447113 3031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.447175 3030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.449488 3033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:32.450031 413 server_base.cc:1048] running on GCE node
I20250626 01:52:32.450325 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:32.450371 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:32.450393 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902752450390 us; error 0 us; skew 500 ppm
I20250626 01:52:32.450491 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:32.451483 3026 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43443
I20250626 01:52:32.451599 3026 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:32.451954 413 webserver.cc:469] Webserver started at http://127.0.103.66:44607/ using document root <none> and password file <none>
I20250626 01:52:32.452049 3026 heartbeater.cc:507] Master 127.0.103.126:43443 requested a full tablet report, sending...
I20250626 01:52:32.452178 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:32.452255 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:32.452369 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:32.453007 2899 ts_manager.cc:194] Registered new tserver with Master: 9acf60da2e9e4a21a1e8a5f88ff07ac4 (127.0.103.65:36413)
I20250626 01:52:32.453094 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "34fbe64191bb4d5bacb80ebef4aa3e60"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.454708 2899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58252
I20250626 01:52:32.456259 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:52:32.457864 3038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.458217 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:52:32.458304 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "34fbe64191bb4d5bacb80ebef4aa3e60"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.458412 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:32.478654 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:32.479290 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:32.480142 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:32.481050 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:32.481109 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.481174 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:32.481216 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.494347 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:42875
I20250626 01:52:32.494366 3100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:42875 every 8 connection(s)
I20250626 01:52:32.495995 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:32.499100 3104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.499590 3105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.499797 3107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:32.500370 413 server_base.cc:1048] running on GCE node
I20250626 01:52:32.500659 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:32.500725 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:32.500808 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902752500803 us; error 0 us; skew 500 ppm
I20250626 01:52:32.500999 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:32.502665 413 webserver.cc:469] Webserver started at http://127.0.103.67:41031/ using document root <none> and password file <none>
I20250626 01:52:32.502946 3101 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43443
I20250626 01:52:32.502981 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:32.503089 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:32.503098 3101 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:32.503250 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:32.503542 3101 heartbeater.cc:507] Master 127.0.103.126:43443 requested a full tablet report, sending...
I20250626 01:52:32.504045 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "dea7430bbe8641a7bd53520538f5aab9"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.504550 2899 ts_manager.cc:194] Registered new tserver with Master: 34fbe64191bb4d5bacb80ebef4aa3e60 (127.0.103.66:42875)
I20250626 01:52:32.505640 2899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58268
I20250626 01:52:32.507208 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:52:32.509086 3112 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.509426 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:52:32.509524 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "dea7430bbe8641a7bd53520538f5aab9"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.509635 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:32.538581 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:32.539184 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:32.539762 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:32.540656 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:32.540715 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.540796 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:32.540835 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.554333 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:38453
I20250626 01:52:32.554376 3174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:38453 every 8 connection(s)
I20250626 01:52:32.556119 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:32.559479 3178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.559583 3179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:32.560158 3181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:32.561219 413 server_base.cc:1048] running on GCE node
I20250626 01:52:32.561513 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:32.561573 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:32.561614 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902752561608 us; error 0 us; skew 500 ppm
I20250626 01:52:32.561776 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:32.563213 3175 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43443
I20250626 01:52:32.563345 3175 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:32.563408 413 webserver.cc:469] Webserver started at http://127.0.103.68:38833/ using document root <none> and password file <none>
I20250626 01:52:32.563638 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:32.563674 3175 heartbeater.cc:507] Master 127.0.103.126:43443 requested a full tablet report, sending...
I20250626 01:52:32.563696 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:32.563866 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:32.564577 2899 ts_manager.cc:194] Registered new tserver with Master: dea7430bbe8641a7bd53520538f5aab9 (127.0.103.67:38453)
I20250626 01:52:32.564595 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "f884fa94146b4da49645dea67546f130"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.565352 2899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58282
I20250626 01:52:32.567219 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 01:52:32.568805 3186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.569130 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:52:32.569237 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "f884fa94146b4da49645dea67546f130"
format_stamp: "Formatted at 2025-06-26 01:52:32 on dist-test-slave-psm5"
I20250626 01:52:32.569344 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:32.588814 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:32.589442 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:32.590027 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:32.590955 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:32.591017 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.591099 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:32.591136 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:32.605535 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:37949
I20250626 01:52:32.605566 3248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:37949 every 8 connection(s)
I20250626 01:52:32.613416 3249 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43443
I20250626 01:52:32.613583 3249 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:32.613962 3249 heartbeater.cc:507] Master 127.0.103.126:43443 requested a full tablet report, sending...
I20250626 01:52:32.614883 2899 ts_manager.cc:194] Registered new tserver with Master: f884fa94146b4da49645dea67546f130 (127.0.103.68:37949)
I20250626 01:52:32.615569 413 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.009070697s
I20250626 01:52:32.615751 2899 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58288
I20250626 01:52:33.456135 3026 heartbeater.cc:499] Master 127.0.103.126:43443 was elected leader, sending a full tablet report...
I20250626 01:52:33.507186 3101 heartbeater.cc:499] Master 127.0.103.126:43443 was elected leader, sending a full tablet report...
I20250626 01:52:33.566619 3175 heartbeater.cc:499] Master 127.0.103.126:43443 was elected leader, sending a full tablet report...
I20250626 01:52:33.617188 3249 heartbeater.cc:499] Master 127.0.103.126:43443 was elected leader, sending a full tablet report...
I20250626 01:52:33.651582 413 test_util.cc:276] Using random seed: -794104971
I20250626 01:52:33.664204 2899 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58306:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250626 01:52:33.694839 3066 tablet_service.cc:1468] Processing CreateTablet for tablet fd13b9818560496484e1b6eff01722fa (DEFAULT_TABLE table=test-workload [id=526a66192c8844fbbd300869e31b98bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:52:33.695390 3066 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd13b9818560496484e1b6eff01722fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:33.696506 3140 tablet_service.cc:1468] Processing CreateTablet for tablet fd13b9818560496484e1b6eff01722fa (DEFAULT_TABLE table=test-workload [id=526a66192c8844fbbd300869e31b98bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:52:33.696903 3140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd13b9818560496484e1b6eff01722fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:33.697304 2991 tablet_service.cc:1468] Processing CreateTablet for tablet fd13b9818560496484e1b6eff01722fa (DEFAULT_TABLE table=test-workload [id=526a66192c8844fbbd300869e31b98bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:52:33.697656 2991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd13b9818560496484e1b6eff01722fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:33.701830 3270 tablet_bootstrap.cc:492] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9: Bootstrap starting.
I20250626 01:52:33.703305 3270 tablet_bootstrap.cc:654] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:33.708271 3271 tablet_bootstrap.cc:492] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: Bootstrap starting.
I20250626 01:52:33.709326 3273 tablet_bootstrap.cc:492] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4: Bootstrap starting.
I20250626 01:52:33.709582 3270 tablet_bootstrap.cc:492] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9: No bootstrap required, opened a new log
I20250626 01:52:33.709741 3270 ts_tablet_manager.cc:1397] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250626 01:52:33.710597 3271 tablet_bootstrap.cc:654] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:33.710508 3270 raft_consensus.cc:357] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.710680 3270 raft_consensus.cc:383] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:33.710776 3270 raft_consensus.cc:738] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dea7430bbe8641a7bd53520538f5aab9, State: Initialized, Role: FOLLOWER
I20250626 01:52:33.710706 3273 tablet_bootstrap.cc:654] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:33.711046 3270 consensus_queue.cc:260] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.712523 3270 ts_tablet_manager.cc:1428] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250626 01:52:33.713076 3271 tablet_bootstrap.cc:492] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: No bootstrap required, opened a new log
I20250626 01:52:33.713217 3271 ts_tablet_manager.cc:1397] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250626 01:52:33.713693 3273 tablet_bootstrap.cc:492] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4: No bootstrap required, opened a new log
I20250626 01:52:33.713819 3273 ts_tablet_manager.cc:1397] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20250626 01:52:33.714000 3271 raft_consensus.cc:357] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.714195 3271 raft_consensus.cc:383] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:33.714253 3271 raft_consensus.cc:738] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34fbe64191bb4d5bacb80ebef4aa3e60, State: Initialized, Role: FOLLOWER
I20250626 01:52:33.714444 3271 consensus_queue.cc:260] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.714524 3273 raft_consensus.cc:357] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.714633 3273 raft_consensus.cc:383] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:33.714718 3273 raft_consensus.cc:738] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9acf60da2e9e4a21a1e8a5f88ff07ac4, State: Initialized, Role: FOLLOWER
I20250626 01:52:33.715037 3273 consensus_queue.cc:260] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.715214 3271 ts_tablet_manager.cc:1428] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:52:33.715828 3273 ts_tablet_manager.cc:1428] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250626 01:52:33.746609 3102 tablet.cc:2378] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:52:33.761915 3277 raft_consensus.cc:491] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:33.762123 3277 raft_consensus.cc:513] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.762889 3277 leader_election.cc:290] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9acf60da2e9e4a21a1e8a5f88ff07ac4 (127.0.103.65:36413), dea7430bbe8641a7bd53520538f5aab9 (127.0.103.67:38453)
I20250626 01:52:33.769126 3001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd13b9818560496484e1b6eff01722fa" candidate_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" is_pre_election: true
I20250626 01:52:33.769245 3150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd13b9818560496484e1b6eff01722fa" candidate_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea7430bbe8641a7bd53520538f5aab9" is_pre_election: true
I20250626 01:52:33.769357 3001 raft_consensus.cc:2466] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34fbe64191bb4d5bacb80ebef4aa3e60 in term 0.
I20250626 01:52:33.769476 3150 raft_consensus.cc:2466] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34fbe64191bb4d5bacb80ebef4aa3e60 in term 0.
I20250626 01:52:33.769784 3040 leader_election.cc:304] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34fbe64191bb4d5bacb80ebef4aa3e60, 9acf60da2e9e4a21a1e8a5f88ff07ac4; no voters:
I20250626 01:52:33.770020 3277 raft_consensus.cc:2802] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:33.770098 3277 raft_consensus.cc:491] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:33.770157 3277 raft_consensus.cc:3058] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:33.772069 3277 raft_consensus.cc:513] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.772616 3277 leader_election.cc:290] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [CANDIDATE]: Term 1 election: Requested vote from peers 9acf60da2e9e4a21a1e8a5f88ff07ac4 (127.0.103.65:36413), dea7430bbe8641a7bd53520538f5aab9 (127.0.103.67:38453)
I20250626 01:52:33.773104 3150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd13b9818560496484e1b6eff01722fa" candidate_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dea7430bbe8641a7bd53520538f5aab9"
I20250626 01:52:33.773104 3001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd13b9818560496484e1b6eff01722fa" candidate_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4"
I20250626 01:52:33.773345 3001 raft_consensus.cc:3058] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:33.773342 3150 raft_consensus.cc:3058] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:33.776096 3001 raft_consensus.cc:2466] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34fbe64191bb4d5bacb80ebef4aa3e60 in term 1.
I20250626 01:52:33.776096 3150 raft_consensus.cc:2466] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34fbe64191bb4d5bacb80ebef4aa3e60 in term 1.
I20250626 01:52:33.776517 3040 leader_election.cc:304] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34fbe64191bb4d5bacb80ebef4aa3e60, 9acf60da2e9e4a21a1e8a5f88ff07ac4; no voters:
I20250626 01:52:33.776787 3277 raft_consensus.cc:2802] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:33.777204 3277 raft_consensus.cc:695] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 1 LEADER]: Becoming Leader. State: Replica: 34fbe64191bb4d5bacb80ebef4aa3e60, State: Running, Role: LEADER
I20250626 01:52:33.777446 3277 consensus_queue.cc:237] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } }
I20250626 01:52:33.780252 2898 catalog_manager.cc:5582] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34fbe64191bb4d5bacb80ebef4aa3e60 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34fbe64191bb4d5bacb80ebef4aa3e60" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42875 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:34.298113 3283 consensus_queue.cc:1035] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9acf60da2e9e4a21a1e8a5f88ff07ac4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:34.303215 3288 consensus_queue.cc:1035] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dea7430bbe8641a7bd53520538f5aab9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:34.331955 2952 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:34.332417 2952 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:34.332521 2952 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:36.332973 2952 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:36.333456 2952 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:36.333563 2952 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:38.334066 2952 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:38.334551 2952 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:38.334651 2952 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:40.335116 2952 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:40.335605 2952 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:40.335700 2952 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:42.336220 2952 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:42.336715 2952 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:42.336817 2952 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:44.337231 2952 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:44.337757 2952 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:44.337879 2952 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:44.856424 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:52:44.869074 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:44.869345 413 tablet_replica.cc:331] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4: stopping tablet replica
I20250626 01:52:44.869504 413 raft_consensus.cc:2241] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:44.869681 413 raft_consensus.cc:2270] T fd13b9818560496484e1b6eff01722fa P 9acf60da2e9e4a21a1e8a5f88ff07ac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:44.885339 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:52:44.888562 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:52:44.898025 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:44.898379 413 tablet_replica.cc:331] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60: stopping tablet replica
I20250626 01:52:44.898597 413 raft_consensus.cc:2241] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:44.898984 413 raft_consensus.cc:2270] T fd13b9818560496484e1b6eff01722fa P 34fbe64191bb4d5bacb80ebef4aa3e60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:44.913043 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:52:44.916085 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:52:44.924887 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:44.925181 413 tablet_replica.cc:331] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9: stopping tablet replica
I20250626 01:52:44.925374 413 raft_consensus.cc:2241] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:44.925565 413 raft_consensus.cc:2270] T fd13b9818560496484e1b6eff01722fa P dea7430bbe8641a7bd53520538f5aab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:44.939548 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:52:44.942948 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:52:44.951329 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:44.964582 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:52:44.967614 413 master.cc:561] Master@127.0.103.126:43443 shutting down...
I20250626 01:52:44.976899 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:44.977187 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:44.986768 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P c769ab463fb44d91b64ffe994dea3291: stopping tablet replica
I20250626 01:52:45.000664 413 master.cc:583] Master@127.0.103.126:43443 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12752 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250626 01:52:45.013453 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:40421
I20250626 01:52:45.014014 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:52:45.017402 413 server_base.cc:1048] running on GCE node
W20250626 01:52:45.017352 3297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.017541 3300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.017387 3298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:45.018383 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:45.018463 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:45.018488 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902765018485 us; error 0 us; skew 500 ppm
I20250626 01:52:45.018627 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:45.025056 413 webserver.cc:469] Webserver started at http://127.0.103.126:44959/ using document root <none> and password file <none>
I20250626 01:52:45.025362 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:45.025439 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:45.025573 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:45.026283 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "31fc41fefa8c4b59bdf5ad88d1c23417"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.029230 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:52:45.031113 3305 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.031525 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:52:45.031625 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "31fc41fefa8c4b59bdf5ad88d1c23417"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.031739 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:45.042256 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:45.042840 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:45.055110 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:40421
I20250626 01:52:45.055130 3356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:40421 every 8 connection(s)
I20250626 01:52:45.056577 3357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:45.061100 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417: Bootstrap starting.
I20250626 01:52:45.062682 3357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:45.064736 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417: No bootstrap required, opened a new log
I20250626 01:52:45.065389 3357 raft_consensus.cc:357] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" member_type: VOTER }
I20250626 01:52:45.065527 3357 raft_consensus.cc:383] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:45.065563 3357 raft_consensus.cc:738] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31fc41fefa8c4b59bdf5ad88d1c23417, State: Initialized, Role: FOLLOWER
I20250626 01:52:45.065728 3357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" member_type: VOTER }
I20250626 01:52:45.065835 3357 raft_consensus.cc:397] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:52:45.065887 3357 raft_consensus.cc:491] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:52:45.065971 3357 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:45.067777 3357 raft_consensus.cc:513] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" member_type: VOTER }
I20250626 01:52:45.068109 3357 leader_election.cc:304] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31fc41fefa8c4b59bdf5ad88d1c23417; no voters:
I20250626 01:52:45.068558 3357 leader_election.cc:290] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:52:45.068711 3360 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:45.069363 3360 raft_consensus.cc:695] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 1 LEADER]: Becoming Leader. State: Replica: 31fc41fefa8c4b59bdf5ad88d1c23417, State: Running, Role: LEADER
I20250626 01:52:45.069579 3357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:52:45.069609 3360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" member_type: VOTER }
I20250626 01:52:45.070322 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31fc41fefa8c4b59bdf5ad88d1c23417. Latest consensus state: current_term: 1 leader_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" member_type: VOTER } }
I20250626 01:52:45.070418 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31fc41fefa8c4b59bdf5ad88d1c23417" member_type: VOTER } }
I20250626 01:52:45.070505 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:45.070509 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:45.072758 3365 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:52:45.074209 3365 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:52:45.074921 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:52:45.077540 3365 catalog_manager.cc:1349] Generated new cluster ID: 59ec90c5f50f469fbe57336b20477dd9
I20250626 01:52:45.077642 3365 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:52:45.093274 3365 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:52:45.094139 3365 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:52:45.110643 3365 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417: Generated new TSK 0
I20250626 01:52:45.110996 3365 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:52:45.140276 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:45.143224 3378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.143297 3379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.143607 3381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:45.143640 413 server_base.cc:1048] running on GCE node
I20250626 01:52:45.144042 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:45.144114 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:45.144138 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902765144135 us; error 0 us; skew 500 ppm
I20250626 01:52:45.144264 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:45.145514 413 webserver.cc:469] Webserver started at http://127.0.103.65:41697/ using document root <none> and password file <none>
I20250626 01:52:45.145722 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:45.145783 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:45.145880 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:45.146515 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "009c903deafb499f9f5fd3ef25f02f93"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.149374 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250626 01:52:45.151216 3386 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.151661 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:52:45.151767 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "009c903deafb499f9f5fd3ef25f02f93"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.151947 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:45.172756 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:45.174242 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:45.174974 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:45.175976 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:45.176040 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.176108 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:45.176132 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.192265 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:40523
I20250626 01:52:45.192289 3448 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:40523 every 8 connection(s)
I20250626 01:52:45.194243 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:45.196988 3453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.197229 3454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.198561 3456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:45.198776 413 server_base.cc:1048] running on GCE node
I20250626 01:52:45.199288 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:45.199354 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:45.199388 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902765199383 us; error 0 us; skew 500 ppm
I20250626 01:52:45.199545 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:45.201007 413 webserver.cc:469] Webserver started at http://127.0.103.66:39657/ using document root <none> and password file <none>
I20250626 01:52:45.201186 3449 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40421
I20250626 01:52:45.201272 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:45.201313 3449 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:45.201356 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:45.201475 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:45.201670 3449 heartbeater.cc:507] Master 127.0.103.126:40421 requested a full tablet report, sending...
I20250626 01:52:45.202252 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "0ee189f51f1243fbb4b12dd02b1c1408"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.202680 3322 ts_manager.cc:194] Registered new tserver with Master: 009c903deafb499f9f5fd3ef25f02f93 (127.0.103.65:40523)
I20250626 01:52:45.204339 3322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43256
I20250626 01:52:45.205596 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:52:45.207410 3461 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.207765 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:52:45.207865 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "0ee189f51f1243fbb4b12dd02b1c1408"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.208022 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:45.224994 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:45.225534 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:45.226089 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:45.226943 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:45.227003 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.227082 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:45.227120 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.241385 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:38913
I20250626 01:52:45.241415 3523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:38913 every 8 connection(s)
I20250626 01:52:45.243026 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:45.245854 3527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.247161 3530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:45.247319 3528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:45.250478 413 server_base.cc:1048] running on GCE node
I20250626 01:52:45.250777 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:45.250850 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:45.250886 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902765250880 us; error 0 us; skew 500 ppm
I20250626 01:52:45.251044 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:45.251169 3524 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40421
I20250626 01:52:45.251309 3524 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:45.251665 3524 heartbeater.cc:507] Master 127.0.103.126:40421 requested a full tablet report, sending...
I20250626 01:52:45.252588 413 webserver.cc:469] Webserver started at http://127.0.103.67:39403/ using document root <none> and password file <none>
I20250626 01:52:45.252784 3322 ts_manager.cc:194] Registered new tserver with Master: 0ee189f51f1243fbb4b12dd02b1c1408 (127.0.103.66:38913)
I20250626 01:52:45.252863 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:45.252965 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:45.253103 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:45.253742 3322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43258
I20250626 01:52:45.253774 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "c3ff5be703064504b364c9352b50b686"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.256417 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20250626 01:52:45.258466 3535 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.258841 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:52:45.258965 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "c3ff5be703064504b364c9352b50b686"
format_stamp: "Formatted at 2025-06-26 01:52:45 on dist-test-slave-psm5"
I20250626 01:52:45.259094 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:45.274611 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:45.275169 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:45.275763 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:45.276697 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:45.276782 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.276912 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:45.276979 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:45.288969 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:46639
I20250626 01:52:45.289002 3597 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:46639 every 8 connection(s)
I20250626 01:52:45.296418 3598 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40421
I20250626 01:52:45.296588 3598 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:45.296972 3598 heartbeater.cc:507] Master 127.0.103.126:40421 requested a full tablet report, sending...
I20250626 01:52:45.297876 3322 ts_manager.cc:194] Registered new tserver with Master: c3ff5be703064504b364c9352b50b686 (127.0.103.67:46639)
I20250626 01:52:45.297932 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007998677s
I20250626 01:52:45.298712 3322 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43270
I20250626 01:52:46.206125 3449 heartbeater.cc:499] Master 127.0.103.126:40421 was elected leader, sending a full tablet report...
I20250626 01:52:46.255016 3524 heartbeater.cc:499] Master 127.0.103.126:40421 was elected leader, sending a full tablet report...
I20250626 01:52:46.300174 3598 heartbeater.cc:499] Master 127.0.103.126:40421 was elected leader, sending a full tablet report...
I20250626 01:52:46.333595 413 test_util.cc:276] Using random seed: -781422955
I20250626 01:52:46.348384 3322 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39488:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:52:46.349203 3322 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:52:46.373140 3563 tablet_service.cc:1468] Processing CreateTablet for tablet e19809eae7314b469ddb1ac1fd2567ac (DEFAULT_TABLE table=test-workload [id=5f06606b9d9a401c8883908687d6a712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:52:46.373812 3563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19809eae7314b469ddb1ac1fd2567ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:46.378904 3489 tablet_service.cc:1468] Processing CreateTablet for tablet e19809eae7314b469ddb1ac1fd2567ac (DEFAULT_TABLE table=test-workload [id=5f06606b9d9a401c8883908687d6a712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:52:46.378962 3414 tablet_service.cc:1468] Processing CreateTablet for tablet e19809eae7314b469ddb1ac1fd2567ac (DEFAULT_TABLE table=test-workload [id=5f06606b9d9a401c8883908687d6a712]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:52:46.379393 3489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19809eae7314b469ddb1ac1fd2567ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:46.379427 3414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e19809eae7314b469ddb1ac1fd2567ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:46.382377 3619 tablet_bootstrap.cc:492] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686: Bootstrap starting.
I20250626 01:52:46.384469 3619 tablet_bootstrap.cc:654] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:46.387475 3622 tablet_bootstrap.cc:492] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408: Bootstrap starting.
I20250626 01:52:46.390168 3622 tablet_bootstrap.cc:654] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:46.393237 3620 tablet_bootstrap.cc:492] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93: Bootstrap starting.
I20250626 01:52:46.394279 3619 tablet_bootstrap.cc:492] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686: No bootstrap required, opened a new log
I20250626 01:52:46.394331 3622 tablet_bootstrap.cc:492] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408: No bootstrap required, opened a new log
I20250626 01:52:46.394469 3619 ts_tablet_manager.cc:1397] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250626 01:52:46.394505 3622 ts_tablet_manager.cc:1397] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250626 01:52:46.395269 3622 raft_consensus.cc:357] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.395331 3619 raft_consensus.cc:357] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.395462 3622 raft_consensus.cc:383] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:46.395604 3619 raft_consensus.cc:383] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:46.395678 3620 tablet_bootstrap.cc:654] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:46.395728 3619 raft_consensus.cc:738] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ff5be703064504b364c9352b50b686, State: Initialized, Role: FOLLOWER
I20250626 01:52:46.395653 3622 raft_consensus.cc:738] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee189f51f1243fbb4b12dd02b1c1408, State: Initialized, Role: FOLLOWER
I20250626 01:52:46.396082 3622 consensus_queue.cc:260] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.396068 3619 consensus_queue.cc:260] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.397034 3622 ts_tablet_manager.cc:1428] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:52:46.397379 3619 ts_tablet_manager.cc:1428] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250626 01:52:46.399864 3620 tablet_bootstrap.cc:492] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93: No bootstrap required, opened a new log
I20250626 01:52:46.400132 3620 ts_tablet_manager.cc:1397] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20250626 01:52:46.400944 3620 raft_consensus.cc:357] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.401108 3620 raft_consensus.cc:383] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:46.401189 3620 raft_consensus.cc:738] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009c903deafb499f9f5fd3ef25f02f93, State: Initialized, Role: FOLLOWER
I20250626 01:52:46.401412 3620 consensus_queue.cc:260] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.402133 3620 ts_tablet_manager.cc:1428] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:46.522497 3625 raft_consensus.cc:491] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:46.522774 3625 raft_consensus.cc:513] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.523715 3625 leader_election.cc:290] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ee189f51f1243fbb4b12dd02b1c1408 (127.0.103.66:38913), 009c903deafb499f9f5fd3ef25f02f93 (127.0.103.65:40523)
I20250626 01:52:46.530929 3499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e19809eae7314b469ddb1ac1fd2567ac" candidate_uuid: "c3ff5be703064504b364c9352b50b686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" is_pre_election: true
I20250626 01:52:46.531217 3424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e19809eae7314b469ddb1ac1fd2567ac" candidate_uuid: "c3ff5be703064504b364c9352b50b686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009c903deafb499f9f5fd3ef25f02f93" is_pre_election: true
I20250626 01:52:46.531224 3499 raft_consensus.cc:2466] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ff5be703064504b364c9352b50b686 in term 0.
I20250626 01:52:46.531476 3424 raft_consensus.cc:2466] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3ff5be703064504b364c9352b50b686 in term 0.
I20250626 01:52:46.531798 3539 leader_election.cc:304] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee189f51f1243fbb4b12dd02b1c1408, c3ff5be703064504b364c9352b50b686; no voters:
I20250626 01:52:46.532105 3625 raft_consensus.cc:2802] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:46.532205 3625 raft_consensus.cc:491] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:46.532298 3625 raft_consensus.cc:3058] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:46.534302 3625 raft_consensus.cc:513] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.535053 3625 leader_election.cc:290] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [CANDIDATE]: Term 1 election: Requested vote from peers 0ee189f51f1243fbb4b12dd02b1c1408 (127.0.103.66:38913), 009c903deafb499f9f5fd3ef25f02f93 (127.0.103.65:40523)
I20250626 01:52:46.535535 3499 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e19809eae7314b469ddb1ac1fd2567ac" candidate_uuid: "c3ff5be703064504b364c9352b50b686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ee189f51f1243fbb4b12dd02b1c1408"
I20250626 01:52:46.535581 3424 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e19809eae7314b469ddb1ac1fd2567ac" candidate_uuid: "c3ff5be703064504b364c9352b50b686" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009c903deafb499f9f5fd3ef25f02f93"
I20250626 01:52:46.535717 3499 raft_consensus.cc:3058] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:46.535748 3424 raft_consensus.cc:3058] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:46.538028 3499 raft_consensus.cc:2466] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ff5be703064504b364c9352b50b686 in term 1.
I20250626 01:52:46.538031 3424 raft_consensus.cc:2466] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3ff5be703064504b364c9352b50b686 in term 1.
I20250626 01:52:46.538540 3539 leader_election.cc:304] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee189f51f1243fbb4b12dd02b1c1408, c3ff5be703064504b364c9352b50b686; no voters:
I20250626 01:52:46.538906 3625 raft_consensus.cc:2802] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:46.539415 3625 raft_consensus.cc:695] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 1 LEADER]: Becoming Leader. State: Replica: c3ff5be703064504b364c9352b50b686, State: Running, Role: LEADER
I20250626 01:52:46.539721 3625 consensus_queue.cc:237] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } }
I20250626 01:52:46.542418 3320 catalog_manager.cc:5582] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3ff5be703064504b364c9352b50b686 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "c3ff5be703064504b364c9352b50b686" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ff5be703064504b364c9352b50b686" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:46.970407 3625 consensus_queue.cc:1035] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ee189f51f1243fbb4b12dd02b1c1408" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:46.975304 3635 consensus_queue.cc:1035] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009c903deafb499f9f5fd3ef25f02f93" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 40523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:47.074137 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:47.074921 3375 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:47.075351 3375 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:47.075476 3375 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:48.076915 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:49.076059 3375 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:49.076653 3375 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:49.076794 3375 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:49.079255 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:50.082149 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:51.077272 3375 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:51.077744 3375 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:51.077980 3375 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:51.085090 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:52.088253 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:53.078487 3375 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:53.078953 3375 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:53.079051 3375 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:53.091786 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:54.094955 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:55.079527 3375 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:55.080024 3375 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:55.080134 3375 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:55.097899 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:56.100381 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:57.080600 3375 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:57.081043 3375 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:52:57.081144 3375 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:57.102764 3370 placement_policy_util.cc:407] tablet e19809eae7314b469ddb1ac1fd2567ac: detected majority of replicas (2 of 3) at location L0
I20250626 01:52:57.622067 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:52:57.638155 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:57.638475 413 tablet_replica.cc:331] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93: stopping tablet replica
I20250626 01:52:57.638659 413 raft_consensus.cc:2241] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:57.638880 413 raft_consensus.cc:2270] T e19809eae7314b469ddb1ac1fd2567ac P 009c903deafb499f9f5fd3ef25f02f93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:57.658993 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:52:57.662269 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:52:57.671527 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:57.671851 413 tablet_replica.cc:331] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408: stopping tablet replica
I20250626 01:52:57.672065 413 raft_consensus.cc:2241] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:52:57.672257 413 raft_consensus.cc:2270] T e19809eae7314b469ddb1ac1fd2567ac P 0ee189f51f1243fbb4b12dd02b1c1408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:57.685966 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:52:57.688966 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:52:57.697384 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:52:57.697639 413 tablet_replica.cc:331] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686: stopping tablet replica
I20250626 01:52:57.697830 413 raft_consensus.cc:2241] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:57.698181 413 raft_consensus.cc:2270] T e19809eae7314b469ddb1ac1fd2567ac P c3ff5be703064504b364c9352b50b686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:57.712071 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:52:57.715061 413 master.cc:561] Master@127.0.103.126:40421 shutting down...
I20250626 01:52:57.723279 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:52:57.723569 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:52:57.723692 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 31fc41fefa8c4b59bdf5ad88d1c23417: stopping tablet replica
I20250626 01:52:57.737665 413 master.cc:583] Master@127.0.103.126:40421 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12735 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20250626 01:52:57.749037 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:34867
I20250626 01:52:57.749652 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:57.753014 3642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.753014 3643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.753990 3645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:57.755143 413 server_base.cc:1048] running on GCE node
I20250626 01:52:57.755414 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:57.755486 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:57.755511 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902777755508 us; error 0 us; skew 500 ppm
I20250626 01:52:57.755647 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:57.763290 413 webserver.cc:469] Webserver started at http://127.0.103.126:35945/ using document root <none> and password file <none>
I20250626 01:52:57.763561 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:57.763653 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:57.763780 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:57.764549 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "618d88e4b2554811b16a814603e1da85"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.768056 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250626 01:52:57.769771 3650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.770224 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:52:57.770323 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "618d88e4b2554811b16a814603e1da85"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.770541 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:57.830379 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:57.830901 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:57.842191 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:34867
I20250626 01:52:57.842211 3701 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:34867 every 8 connection(s)
I20250626 01:52:57.843494 3702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:57.847540 3702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85: Bootstrap starting.
I20250626 01:52:57.848814 3702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:57.850412 3702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85: No bootstrap required, opened a new log
I20250626 01:52:57.851038 3702 raft_consensus.cc:357] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618d88e4b2554811b16a814603e1da85" member_type: VOTER }
I20250626 01:52:57.851179 3702 raft_consensus.cc:383] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:57.851233 3702 raft_consensus.cc:738] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618d88e4b2554811b16a814603e1da85, State: Initialized, Role: FOLLOWER
I20250626 01:52:57.851441 3702 consensus_queue.cc:260] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618d88e4b2554811b16a814603e1da85" member_type: VOTER }
I20250626 01:52:57.851549 3702 raft_consensus.cc:397] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:52:57.851609 3702 raft_consensus.cc:491] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:52:57.851686 3702 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:57.853477 3702 raft_consensus.cc:513] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618d88e4b2554811b16a814603e1da85" member_type: VOTER }
I20250626 01:52:57.853653 3702 leader_election.cc:304] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 618d88e4b2554811b16a814603e1da85; no voters:
I20250626 01:52:57.854086 3702 leader_election.cc:290] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:52:57.854224 3705 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:57.854753 3705 raft_consensus.cc:695] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 1 LEADER]: Becoming Leader. State: Replica: 618d88e4b2554811b16a814603e1da85, State: Running, Role: LEADER
I20250626 01:52:57.854925 3702 sys_catalog.cc:564] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:52:57.855010 3705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618d88e4b2554811b16a814603e1da85" member_type: VOTER }
I20250626 01:52:57.855908 3706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "618d88e4b2554811b16a814603e1da85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618d88e4b2554811b16a814603e1da85" member_type: VOTER } }
I20250626 01:52:57.855932 3707 sys_catalog.cc:455] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 618d88e4b2554811b16a814603e1da85. Latest consensus state: current_term: 1 leader_uuid: "618d88e4b2554811b16a814603e1da85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "618d88e4b2554811b16a814603e1da85" member_type: VOTER } }
I20250626 01:52:57.856129 3706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:57.856240 3707 sys_catalog.cc:458] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [sys.catalog]: This master's current role is: LEADER
I20250626 01:52:57.857242 3710 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:52:57.858702 3710 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:52:57.860126 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:52:57.862010 3710 catalog_manager.cc:1349] Generated new cluster ID: 08258bf623104786b0ba29328e4044db
I20250626 01:52:57.862097 3710 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:52:57.870086 3710 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:52:57.870862 3710 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:52:57.889781 3710 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85: Generated new TSK 0
I20250626 01:52:57.890045 3710 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:52:57.892964 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:57.895938 3724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.895946 3723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.896420 3726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:57.896855 413 server_base.cc:1048] running on GCE node
I20250626 01:52:57.897161 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:57.897221 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:57.897261 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902777897257 us; error 0 us; skew 500 ppm
I20250626 01:52:57.897436 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:57.898852 413 webserver.cc:469] Webserver started at http://127.0.103.65:42821/ using document root <none> and password file <none>
I20250626 01:52:57.899075 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:57.899150 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:57.899282 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:57.899995 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.902604 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:57.904323 3731 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.904716 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:52:57.904819 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.904961 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:57.922439 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:57.923002 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:57.923571 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:57.924500 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:57.924561 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.924660 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:57.924707 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.938184 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:34305
I20250626 01:52:57.938212 3793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:34305 every 8 connection(s)
I20250626 01:52:57.939774 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:57.942701 3798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.943158 3799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:57.943624 413 server_base.cc:1048] running on GCE node
W20250626 01:52:57.943665 3801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:57.944182 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:57.944242 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:57.944267 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902777944264 us; error 0 us; skew 500 ppm
I20250626 01:52:57.944406 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:57.946235 413 webserver.cc:469] Webserver started at http://127.0.103.66:42577/ using document root <none> and password file <none>
I20250626 01:52:57.946466 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:57.946520 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:57.946600 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:57.947130 3794 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34867
I20250626 01:52:57.947244 3794 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:57.947256 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "311ee4b8be9f47a8a3de56840e44425a"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.947525 3794 heartbeater.cc:507] Master 127.0.103.126:34867 requested a full tablet report, sending...
I20250626 01:52:57.948474 3667 ts_manager.cc:194] Registered new tserver with Master: 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:52:57.949939 3667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55126
I20250626 01:52:57.950193 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250626 01:52:57.952064 3806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.952385 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:52:57.952479 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "311ee4b8be9f47a8a3de56840e44425a"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.952591 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:57.968358 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:57.968914 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:57.969511 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:57.970288 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:57.970346 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.970430 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:57.970467 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.983157 3868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:40297 every 8 connection(s)
I20250626 01:52:57.983157 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:40297
I20250626 01:52:57.984822 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:52:57.987691 3873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.987773 3872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:52:57.988051 3875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:52:57.989161 413 server_base.cc:1048] running on GCE node
I20250626 01:52:57.989490 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:52:57.989547 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:52:57.989609 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902777989604 us; error 0 us; skew 500 ppm
I20250626 01:52:57.989756 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:52:57.991189 3869 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34867
I20250626 01:52:57.991310 413 webserver.cc:469] Webserver started at http://127.0.103.67:39043/ using document root <none> and password file <none>
I20250626 01:52:57.991331 3869 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:57.991557 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:52:57.991631 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:52:57.991698 3869 heartbeater.cc:507] Master 127.0.103.126:34867 requested a full tablet report, sending...
I20250626 01:52:57.991748 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:52:57.992499 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "001b6ec9d9bc46c79e57a4915c44c925"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.992692 3667 ts_manager.cc:194] Registered new tserver with Master: 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:57.993767 3667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55128
I20250626 01:52:57.995440 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250626 01:52:57.997015 3880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:57.997371 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:52:57.997457 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "001b6ec9d9bc46c79e57a4915c44c925"
format_stamp: "Formatted at 2025-06-26 01:52:57 on dist-test-slave-psm5"
I20250626 01:52:57.997562 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:52:58.009159 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:52:58.009729 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:52:58.010282 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:52:58.011071 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:52:58.011129 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:58.011211 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:52:58.011253 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:52:58.023192 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35233
I20250626 01:52:58.023211 3942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35233 every 8 connection(s)
I20250626 01:52:58.029896 3943 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34867
I20250626 01:52:58.030032 3943 heartbeater.cc:461] Registering TS with master...
I20250626 01:52:58.030351 3943 heartbeater.cc:507] Master 127.0.103.126:34867 requested a full tablet report, sending...
I20250626 01:52:58.031196 3667 ts_manager.cc:194] Registered new tserver with Master: 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:58.031656 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006848731s
I20250626 01:52:58.032021 3667 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55136
I20250626 01:52:58.951587 3794 heartbeater.cc:499] Master 127.0.103.126:34867 was elected leader, sending a full tablet report...
I20250626 01:52:58.995074 3869 heartbeater.cc:499] Master 127.0.103.126:34867 was elected leader, sending a full tablet report...
I20250626 01:52:59.033418 3943 heartbeater.cc:499] Master 127.0.103.126:34867 was elected leader, sending a full tablet report...
I20250626 01:52:59.060135 413 test_util.cc:276] Using random seed: -768696417
I20250626 01:52:59.075872 3667 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55146:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:52:59.076802 3667 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:52:59.111548 3908 tablet_service.cc:1468] Processing CreateTablet for tablet f43f00d4508b48278aeea9fb37615694 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250626 01:52:59.111871 3905 tablet_service.cc:1468] Processing CreateTablet for tablet 16e2c24dbd4f446caf8d09565cbac048 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250626 01:52:59.112133 3908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f00d4508b48278aeea9fb37615694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.112468 3904 tablet_service.cc:1468] Processing CreateTablet for tablet 71684e4b73c24f57b67e052454de9097 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250626 01:52:59.112859 3904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71684e4b73c24f57b67e052454de9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.115084 3906 tablet_service.cc:1468] Processing CreateTablet for tablet bc9d4e02a1e14b369f999b13dad44544 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250626 01:52:59.115486 3906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d4e02a1e14b369f999b13dad44544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.116211 3903 tablet_service.cc:1468] Processing CreateTablet for tablet 96fb20902f87437487c5f36d71de0b9b (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250626 01:52:59.116813 3902 tablet_service.cc:1468] Processing CreateTablet for tablet 346b250db608442d84f96d2c2e603775 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250626 01:52:59.117341 3901 tablet_service.cc:1468] Processing CreateTablet for tablet 978023fb60024f26a213be77e43d8140 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250626 01:52:59.117939 3900 tablet_service.cc:1468] Processing CreateTablet for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250626 01:52:59.118520 3899 tablet_service.cc:1468] Processing CreateTablet for tablet e80ca9ec9a27407fa4e5ae542fdad35d (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250626 01:52:59.111548 3907 tablet_service.cc:1468] Processing CreateTablet for tablet eec015ad4dee46efb54559801ff9896b (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250626 01:52:59.119148 3905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16e2c24dbd4f446caf8d09565cbac048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.119851 3903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fb20902f87437487c5f36d71de0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.121255 3759 tablet_service.cc:1468] Processing CreateTablet for tablet f43f00d4508b48278aeea9fb37615694 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250626 01:52:59.121405 3834 tablet_service.cc:1468] Processing CreateTablet for tablet f43f00d4508b48278aeea9fb37615694 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION VALUES < 178956970
I20250626 01:52:59.121625 3759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f00d4508b48278aeea9fb37615694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.121793 3834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f00d4508b48278aeea9fb37615694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.122239 3900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.122931 3833 tablet_service.cc:1468] Processing CreateTablet for tablet eec015ad4dee46efb54559801ff9896b (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250626 01:52:59.123317 3833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec015ad4dee46efb54559801ff9896b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.128386 3832 tablet_service.cc:1468] Processing CreateTablet for tablet bc9d4e02a1e14b369f999b13dad44544 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250626 01:52:59.129060 3832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d4e02a1e14b369f999b13dad44544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.129184 3757 tablet_service.cc:1468] Processing CreateTablet for tablet bc9d4e02a1e14b369f999b13dad44544 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20250626 01:52:59.129570 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d4e02a1e14b369f999b13dad44544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.132346 3831 tablet_service.cc:1468] Processing CreateTablet for tablet 16e2c24dbd4f446caf8d09565cbac048 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250626 01:52:59.132846 3831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16e2c24dbd4f446caf8d09565cbac048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.136240 3830 tablet_service.cc:1468] Processing CreateTablet for tablet 71684e4b73c24f57b67e052454de9097 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250626 01:52:59.136528 3829 tablet_service.cc:1468] Processing CreateTablet for tablet 96fb20902f87437487c5f36d71de0b9b (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250626 01:52:59.136706 3830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71684e4b73c24f57b67e052454de9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.136924 3829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fb20902f87437487c5f36d71de0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.140347 3828 tablet_service.cc:1468] Processing CreateTablet for tablet 346b250db608442d84f96d2c2e603775 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250626 01:52:59.140844 3828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346b250db608442d84f96d2c2e603775. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.136240 3757 tablet_service.cc:1468] Processing CreateTablet for tablet 16e2c24dbd4f446caf8d09565cbac048 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20250626 01:52:59.144055 3757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16e2c24dbd4f446caf8d09565cbac048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.144299 3827 tablet_service.cc:1468] Processing CreateTablet for tablet 978023fb60024f26a213be77e43d8140 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250626 01:52:59.144691 3827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978023fb60024f26a213be77e43d8140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.148347 3826 tablet_service.cc:1468] Processing CreateTablet for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250626 01:52:59.148913 3826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.149143 3756 tablet_service.cc:1468] Processing CreateTablet for tablet 71684e4b73c24f57b67e052454de9097 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20250626 01:52:59.149559 3756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71684e4b73c24f57b67e052454de9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.152307 3825 tablet_service.cc:1468] Processing CreateTablet for tablet e80ca9ec9a27407fa4e5ae542fdad35d (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250626 01:52:59.152835 3825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80ca9ec9a27407fa4e5ae542fdad35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.128465 3758 tablet_service.cc:1468] Processing CreateTablet for tablet eec015ad4dee46efb54559801ff9896b (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20250626 01:52:59.156203 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec015ad4dee46efb54559801ff9896b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.156893 3756 tablet_service.cc:1468] Processing CreateTablet for tablet 96fb20902f87437487c5f36d71de0b9b (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20250626 01:52:59.157298 3756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fb20902f87437487c5f36d71de0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.160126 3899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80ca9ec9a27407fa4e5ae542fdad35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.160832 3902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346b250db608442d84f96d2c2e603775. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.161387 3901 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978023fb60024f26a213be77e43d8140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.161326 3755 tablet_service.cc:1468] Processing CreateTablet for tablet 346b250db608442d84f96d2c2e603775 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20250626 01:52:59.161841 3755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 346b250db608442d84f96d2c2e603775. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.161870 3907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec015ad4dee46efb54559801ff9896b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.167786 3908 tablet_service.cc:1468] Processing CreateTablet for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250626 01:52:59.168313 3908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ad5196a1fa4032afaa62ccab7e3fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.173511 3907 tablet_service.cc:1468] Processing CreateTablet for tablet 34f950fcb4ee4d5aad0d3f06ed2081af (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250626 01:52:59.174079 3907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f950fcb4ee4d5aad0d3f06ed2081af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.183571 3755 tablet_service.cc:1468] Processing CreateTablet for tablet 978023fb60024f26a213be77e43d8140 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20250626 01:52:59.184212 3755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978023fb60024f26a213be77e43d8140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.189069 3754 tablet_service.cc:1468] Processing CreateTablet for tablet e80ca9ec9a27407fa4e5ae542fdad35d (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20250626 01:52:59.189632 3754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80ca9ec9a27407fa4e5ae542fdad35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.192305 3826 tablet_service.cc:1468] Processing CreateTablet for tablet 34f950fcb4ee4d5aad0d3f06ed2081af (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250626 01:52:59.204521 3826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f950fcb4ee4d5aad0d3f06ed2081af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.203797 3758 tablet_service.cc:1468] Processing CreateTablet for tablet 34f950fcb4ee4d5aad0d3f06ed2081af (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20250626 01:52:59.205937 3758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f950fcb4ee4d5aad0d3f06ed2081af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.197060 3754 tablet_service.cc:1468] Processing CreateTablet for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250626 01:52:59.208274 3754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ad5196a1fa4032afaa62ccab7e3fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.211442 3756 tablet_service.cc:1468] Processing CreateTablet for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20250626 01:52:59.148347 3830 tablet_service.cc:1468] Processing CreateTablet for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 (DEFAULT_TABLE table=test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20250626 01:52:59.212786 3830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ad5196a1fa4032afaa62ccab7e3fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.220292 3756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:52:59.224429 3964 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.226387 3964 tablet_bootstrap.cc:654] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.228914 3963 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.229408 3965 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.231842 3963 tablet_bootstrap.cc:654] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.231998 3965 tablet_bootstrap.cc:654] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.232962 3964 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.233124 3964 ts_tablet_manager.cc:1397] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250626 01:52:59.234539 3964 raft_consensus.cc:357] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.234691 3964 raft_consensus.cc:383] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.234783 3964 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.234927 3963 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.234927 3965 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.235087 3965 ts_tablet_manager.cc:1397] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250626 01:52:59.235087 3963 ts_tablet_manager.cc:1397] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250626 01:52:59.235028 3964 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.235947 3963 raft_consensus.cc:357] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.235947 3965 raft_consensus.cc:357] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.236147 3963 raft_consensus.cc:383] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.236156 3965 raft_consensus.cc:383] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.236274 3963 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.236302 3965 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.236510 3964 ts_tablet_manager.cc:1428] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250626 01:52:59.236522 3963 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.236505 3965 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.236829 3964 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.237243 3963 ts_tablet_manager.cc:1428] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:59.237491 3963 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.239439 3963 tablet_bootstrap.cc:654] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.240415 3964 tablet_bootstrap.cc:654] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.242264 3964 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.242405 3964 ts_tablet_manager.cc:1397] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250626 01:52:59.243078 3964 raft_consensus.cc:357] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.243227 3964 raft_consensus.cc:383] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.243307 3964 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.243518 3964 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.244235 3964 ts_tablet_manager.cc:1428] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:59.244568 3964 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.246261 3964 tablet_bootstrap.cc:654] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.247923 3965 ts_tablet_manager.cc:1428] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.013s user 0.004s sys 0.005s
I20250626 01:52:59.249540 3965 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.252746 3965 tablet_bootstrap.cc:654] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.255702 3963 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.255702 3965 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.256063 3963 ts_tablet_manager.cc:1397] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.008s
I20250626 01:52:59.256098 3965 ts_tablet_manager.cc:1397] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250626 01:52:59.256807 3965 raft_consensus.cc:357] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.256867 3963 raft_consensus.cc:357] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.256968 3965 raft_consensus.cc:383] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.256983 3964 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.257062 3965 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.257012 3963 raft_consensus.cc:383] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.257143 3963 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.257190 3964 ts_tablet_manager.cc:1397] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.010s
I20250626 01:52:59.257323 3965 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.257407 3963 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.257825 3965 ts_tablet_manager.cc:1428] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:52:59.257941 3963 ts_tablet_manager.cc:1428] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:52:59.258148 3965 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.258219 3963 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.258189 3964 raft_consensus.cc:357] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.258353 3964 raft_consensus.cc:383] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.258421 3964 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.258669 3964 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.259866 3963 tablet_bootstrap.cc:654] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.259904 3965 tablet_bootstrap.cc:654] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.259912 3964 ts_tablet_manager.cc:1428] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250626 01:52:59.260324 3964 tablet_bootstrap.cc:492] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.262380 3964 tablet_bootstrap.cc:654] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.265201 3965 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.265437 3965 ts_tablet_manager.cc:1397] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.003s
I20250626 01:52:59.266337 3965 raft_consensus.cc:357] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.266614 3965 raft_consensus.cc:383] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.266727 3965 raft_consensus.cc:738] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.267037 3965 consensus_queue.cc:260] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.267648 3965 ts_tablet_manager.cc:1428] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:52:59.268437 3965 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.270889 3965 tablet_bootstrap.cc:654] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.273273 3963 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.273582 3963 ts_tablet_manager.cc:1397] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.013s
I20250626 01:52:59.274716 3963 raft_consensus.cc:357] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.275044 3963 raft_consensus.cc:383] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.275169 3963 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.275473 3963 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.276357 3963 ts_tablet_manager.cc:1428] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:52:59.276513 3964 tablet_bootstrap.cc:492] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.276685 3964 ts_tablet_manager.cc:1397] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.010s
I20250626 01:52:59.276712 3963 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.277247 3965 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.277451 3965 ts_tablet_manager.cc:1397] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20250626 01:52:59.277455 3964 raft_consensus.cc:357] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.277794 3964 raft_consensus.cc:383] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.277894 3964 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.278158 3964 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.278282 3965 raft_consensus.cc:357] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.278394 3963 tablet_bootstrap.cc:654] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.278488 3965 raft_consensus.cc:383] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.278610 3965 raft_consensus.cc:738] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.278848 3964 ts_tablet_manager.cc:1428] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:52:59.278924 3965 consensus_queue.cc:260] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.279152 3964 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.279502 3965 ts_tablet_manager.cc:1428] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:52:59.280269 3965 tablet_bootstrap.cc:492] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.281251 3964 tablet_bootstrap.cc:654] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.281781 3965 tablet_bootstrap.cc:654] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.290290 3963 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.290560 3963 ts_tablet_manager.cc:1397] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20250626 01:52:59.290712 3964 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.290756 3965 tablet_bootstrap.cc:492] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.290892 3964 ts_tablet_manager.cc:1397] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250626 01:52:59.291002 3965 ts_tablet_manager.cc:1397] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20250626 01:52:59.291530 3964 raft_consensus.cc:357] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.291466 3963 raft_consensus.cc:357] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.291682 3964 raft_consensus.cc:383] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.291718 3963 raft_consensus.cc:383] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.291596 3965 raft_consensus.cc:357] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.291790 3964 raft_consensus.cc:738] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.291792 3963 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.291852 3965 raft_consensus.cc:383] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.291988 3965 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.292054 3963 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.292196 3965 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.292541 3963 ts_tablet_manager.cc:1428] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:52:59.292676 3965 ts_tablet_manager.cc:1428] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:52:59.292797 3963 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.292932 3965 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.294407 3965 tablet_bootstrap.cc:654] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.292054 3964 consensus_queue.cc:260] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.294924 3964 ts_tablet_manager.cc:1428] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250626 01:52:59.295317 3964 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.294417 3963 tablet_bootstrap.cc:654] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.297837 3964 tablet_bootstrap.cc:654] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.300237 3965 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.300446 3965 ts_tablet_manager.cc:1397] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250626 01:52:59.301515 3965 raft_consensus.cc:357] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.301752 3965 raft_consensus.cc:383] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.301843 3965 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.302098 3965 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.302842 3965 ts_tablet_manager.cc:1428] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:52:59.303207 3965 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.306157 3965 tablet_bootstrap.cc:654] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.306564 3964 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.306788 3964 ts_tablet_manager.cc:1397] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250626 01:52:59.307734 3964 raft_consensus.cc:357] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.308149 3964 raft_consensus.cc:383] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.308285 3963 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.308303 3964 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.308441 3963 ts_tablet_manager.cc:1397] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20250626 01:52:59.308713 3964 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.309087 3963 raft_consensus.cc:357] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.309301 3963 raft_consensus.cc:383] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.309397 3963 raft_consensus.cc:738] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.309548 3964 ts_tablet_manager.cc:1428] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:52:59.309638 3963 consensus_queue.cc:260] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.310240 3964 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.311347 3963 ts_tablet_manager.cc:1428] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:52:59.311686 3963 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.312810 3964 tablet_bootstrap.cc:654] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.313578 3963 tablet_bootstrap.cc:654] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.315495 3965 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.315637 3965 ts_tablet_manager.cc:1397] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250626 01:52:59.316089 3963 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.316222 3963 ts_tablet_manager.cc:1397] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250626 01:52:59.316411 3965 raft_consensus.cc:357] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.316582 3965 raft_consensus.cc:383] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.316674 3965 raft_consensus.cc:738] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.316926 3965 consensus_queue.cc:260] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.317000 3963 raft_consensus.cc:357] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.317210 3963 raft_consensus.cc:383] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.317333 3963 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.317430 3965 ts_tablet_manager.cc:1428] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:59.317549 3963 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.317798 3965 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.318192 3963 ts_tablet_manager.cc:1428] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:59.318488 3963 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.320004 3965 tablet_bootstrap.cc:654] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.320856 3963 tablet_bootstrap.cc:654] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.322186 3965 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.322343 3965 ts_tablet_manager.cc:1397] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250626 01:52:59.322983 3965 raft_consensus.cc:357] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.323165 3965 raft_consensus.cc:383] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.323261 3965 raft_consensus.cc:738] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.323510 3965 consensus_queue.cc:260] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.324062 3965 ts_tablet_manager.cc:1428] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:52:59.324330 3965 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.324983 3970 raft_consensus.cc:491] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.325229 3970 raft_consensus.cc:513] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.326007 3965 tablet_bootstrap.cc:654] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.326505 3970 leader_election.cc:290] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.333791 3964 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.334053 3964 ts_tablet_manager.cc:1397] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20250626 01:52:59.335366 3964 raft_consensus.cc:357] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.335681 3964 raft_consensus.cc:383] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.335789 3964 raft_consensus.cc:738] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.337003 3964 consensus_queue.cc:260] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.337940 3964 ts_tablet_manager.cc:1428] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250626 01:52:59.338479 3964 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.342357 3964 tablet_bootstrap.cc:654] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.345993 3964 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.346205 3964 ts_tablet_manager.cc:1397] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20250626 01:52:59.347091 3964 raft_consensus.cc:357] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.347316 3964 raft_consensus.cc:383] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.347390 3964 raft_consensus.cc:738] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.347633 3964 consensus_queue.cc:260] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.348284 3964 ts_tablet_manager.cc:1428] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250626 01:52:59.348661 3964 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.349908 3971 raft_consensus.cc:491] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.350172 3971 raft_consensus.cc:513] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.350924 3964 tablet_bootstrap.cc:654] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.351817 3971 leader_election.cc:290] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.362324 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.362824 3769 raft_consensus.cc:2466] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.363207 3963 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.363996 3963 ts_tablet_manager.cc:1397] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.021s
I20250626 01:52:59.365012 3963 raft_consensus.cc:357] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.366044 3964 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.366150 3963 raft_consensus.cc:383] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.366461 3965 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.366690 3963 raft_consensus.cc:738] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.366938 3884 leader_election.cc:304] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.367031 3965 ts_tablet_manager.cc:1397] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.018s
I20250626 01:52:59.368996 3965 raft_consensus.cc:357] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.369233 3965 raft_consensus.cc:383] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.369333 3965 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.367024 3963 consensus_queue.cc:260] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.369593 3965 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.369923 3963 ts_tablet_manager.cc:1428] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250626 01:52:59.370115 3965 ts_tablet_manager.cc:1428] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:52:59.370241 3963 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.371359 3965 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.372476 3963 tablet_bootstrap.cc:654] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.367165 3964 ts_tablet_manager.cc:1397] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.013s
I20250626 01:52:59.373670 3965 tablet_bootstrap.cc:654] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.374888 3964 raft_consensus.cc:357] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.375121 3964 raft_consensus.cc:383] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.375229 3964 raft_consensus.cc:738] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.375480 3964 consensus_queue.cc:260] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.375645 3969 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.376030 3970 raft_consensus.cc:2802] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.376152 3970 raft_consensus.cc:491] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.376269 3970 raft_consensus.cc:3058] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.376338 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.376634 3844 raft_consensus.cc:2466] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.377246 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.380424 3970 raft_consensus.cc:513] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:52:59.381426 3809 leader_election.cc:343] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.375795 3969 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.382023 4002 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.382213 4002 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.383005 3969 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.385641 3970 leader_election.cc:290] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.385993 3970 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.386242 3970 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.383008 4002 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.389204 3970 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:52:59.390204 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.390483 3769 raft_consensus.cc:3058] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.395270 3769 raft_consensus.cc:2466] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.395205 3768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.396510 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
W20250626 01:52:59.396709 3810 leader_election.cc:343] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.396829 3769 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.396873 3810 leader_election.cc:304] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a; no voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:52:59.397348 3971 raft_consensus.cc:2747] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 01:52:59.397747 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.398010 3844 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.399127 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.399377 3844 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 0.
I20250626 01:52:59.399816 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.400053 3735 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.400107 3844 raft_consensus.cc:3058] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.400368 3969 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.400507 3969 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.400630 3969 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.401777 3964 ts_tablet_manager.cc:1428] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.028s user 0.011s sys 0.003s
I20250626 01:52:59.402216 3963 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.402411 3963 ts_tablet_manager.cc:1397] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.008s
I20250626 01:52:59.403268 3963 raft_consensus.cc:357] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.403456 3963 raft_consensus.cc:383] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.403417 3969 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.403532 3963 raft_consensus.cc:738] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.403816 3844 raft_consensus.cc:2466] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.403779 3963 consensus_queue.cc:260] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.404402 3963 ts_tablet_manager.cc:1428] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:52:59.405139 3969 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.402197 3964 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.405791 3963 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.405581 3843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.409049 3843 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.409088 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.409567 3844 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.410029 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.410226 3769 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.410396 3965 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.410523 3963 tablet_bootstrap.cc:654] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.410688 3965 ts_tablet_manager.cc:1397] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.009s
I20250626 01:52:59.411175 3884 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.411559 4003 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.411711 4003 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.411960 4003 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.411967 3965 raft_consensus.cc:357] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.416244 3965 raft_consensus.cc:383] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.416651 3884 leader_election.cc:304] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.417232 3970 raft_consensus.cc:2802] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.417433 3970 raft_consensus.cc:695] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.417834 3970 consensus_queue.cc:237] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.420125 3965 raft_consensus.cc:738] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.420375 4003 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.420492 3843 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 1.
I20250626 01:52:59.420666 3884 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.421253 3735 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.421653 3965 consensus_queue.cc:260] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.423660 3969 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.423846 3969 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:52:59.424168 3969 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.425549 3965 ts_tablet_manager.cc:1428] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250626 01:52:59.426146 3965 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.422065 4003 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.426761 4002 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.429103 3843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.429360 4002 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.429826 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.431246 3843 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.431475 4002 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.432261 3769 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.432381 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.432659 3918 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 0.
I20250626 01:52:59.433331 3971 raft_consensus.cc:491] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.433545 3971 raft_consensus.cc:513] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.435712 3768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.435856 3971 leader_election.cc:290] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.435642 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:52:59.436403 3918 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.436527 3768 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.436537 3963 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.438776 3810 leader_election.cc:304] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.439327 3963 ts_tablet_manager.cc:1397] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.007s
I20250626 01:52:59.442269 3971 raft_consensus.cc:2802] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.443979 3963 raft_consensus.cc:357] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.444418 3963 raft_consensus.cc:383] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.444589 3963 raft_consensus.cc:738] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.444814 3971 raft_consensus.cc:491] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.444972 3971 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.445091 3963 consensus_queue.cc:260] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.443559 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.448449 3918 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 1.
I20250626 01:52:59.448575 3917 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.449764 3963 ts_tablet_manager.cc:1428] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250626 01:52:59.449828 3964 tablet_bootstrap.cc:654] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.448700 3665 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.450546 3963 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.456360 3843 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.456913 3884 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.457500 3971 raft_consensus.cc:513] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.458953 3969 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.456625 3667 catalog_manager.cc:5582] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:59.460064 3969 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.460731 3768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.461004 3768 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.461542 3970 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.460744 3969 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.461282 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.462507 3769 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.463300 3971 leader_election.cc:290] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.463626 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:52:59.463856 3970 raft_consensus.cc:695] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.464236 3843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
W20250626 01:52:59.464273 3734 leader_election.cc:343] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.469358 3971 raft_consensus.cc:491] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.469858 3971 raft_consensus.cc:513] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.470088 3917 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.471215 3843 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 0.
I20250626 01:52:59.471657 3970 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.472048 3971 leader_election.cc:290] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.472436 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.472862 3735 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters: 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:52:59.473268 4002 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.473498 3969 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.473640 3969 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.473771 3969 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.474148 4002 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:52:59.477159 3768 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.478077 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.478404 3769 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.478859 3917 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.479743 3768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.480011 3768 raft_consensus.cc:2466] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.480505 3810 leader_election.cc:304] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.480569 3963 tablet_bootstrap.cc:654] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.480890 3810 leader_election.cc:304] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.481000 3964 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.481211 3964 ts_tablet_manager.cc:1397] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.005s
I20250626 01:52:59.481395 3971 raft_consensus.cc:2802] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.481521 3843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.481710 3769 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.481796 4010 raft_consensus.cc:2802] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.481761 3843 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.481912 4010 raft_consensus.cc:491] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.482029 4010 raft_consensus.cc:3058] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.482074 3964 raft_consensus.cc:357] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.482296 3964 raft_consensus.cc:383] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.482406 3964 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
W20250626 01:52:59.482424 3809 leader_election.cc:343] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.482728 3964 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.483028 3884 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.484567 3969 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.485749 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.485947 3844 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.486394 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
W20250626 01:52:59.487257 3734 leader_election.cc:343] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.487517 3969 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.489426 4011 raft_consensus.cc:491] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.489714 3844 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 1.
I20250626 01:52:59.489616 4011 raft_consensus.cc:513] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.490001 3843 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.489585 4002 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.490399 4002 raft_consensus.cc:695] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.490608 3735 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters: 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:52:59.490697 4002 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.490983 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.491072 3969 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.491217 3917 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.491221 3969 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:52:59.491652 3969 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.491791 3965 tablet_bootstrap.cc:654] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.491976 3971 raft_consensus.cc:695] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:52:59.493144 3809 leader_election.cc:304] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.493239 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.493419 3769 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.493945 3971 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.492523 4010 raft_consensus.cc:513] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.496541 4010 leader_election.cc:290] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.496762 4010 raft_consensus.cc:2802] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.496861 4010 raft_consensus.cc:491] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.496971 4010 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.497074 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
W20250626 01:52:59.497594 3809 leader_election.cc:343] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.499198 3667 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:59.499505 3964 ts_tablet_manager.cc:1428] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.018s user 0.001s sys 0.002s
I20250626 01:52:59.500358 3964 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.502081 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.502298 3769 raft_consensus.cc:3058] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.503741 3665 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 0 to 1, leader changed from <none> to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.506034 3965 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.506352 3965 ts_tablet_manager.cc:1397] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.080s user 0.006s sys 0.000s
I20250626 01:52:59.506700 3769 raft_consensus.cc:2466] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.507365 3810 leader_election.cc:304] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters: 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:52:59.507723 3971 raft_consensus.cc:2802] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.508008 3971 raft_consensus.cc:695] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:52:59.508378 3971 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.509339 3964 tablet_bootstrap.cc:654] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.507598 3965 raft_consensus.cc:357] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.510787 3965 raft_consensus.cc:383] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.510967 3965 raft_consensus.cc:738] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.511397 3965 consensus_queue.cc:260] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.511543 3665 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.512336 3965 ts_tablet_manager.cc:1428] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250626 01:52:59.512648 3965 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: Bootstrap starting.
I20250626 01:52:59.513329 3963 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.513582 3963 ts_tablet_manager.cc:1397] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.009s
I20250626 01:52:59.513949 4010 raft_consensus.cc:513] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.514609 4002 raft_consensus.cc:491] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.514828 4002 raft_consensus.cc:513] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.515241 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.515443 3769 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.515480 4002 leader_election.cc:290] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.514662 3965 tablet_bootstrap.cc:654] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.516424 3768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
W20250626 01:52:59.517144 3884 leader_election.cc:343] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): Illegal state: must be running to vote when last-logged opid is not known
I20250626 01:52:59.517489 3963 raft_consensus.cc:357] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.517787 3963 raft_consensus.cc:383] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.517959 3963 raft_consensus.cc:738] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.518350 3963 consensus_queue.cc:260] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.518288 3667 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 1 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.518664 3769 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.518642 4011 leader_election.cc:290] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.519047 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.519232 3844 raft_consensus.cc:2466] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.520283 3810 leader_election.cc:304] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.520253 3884 leader_election.cc:304] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters: 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:52:59.520617 4011 raft_consensus.cc:2802] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.521117 4010 leader_election.cc:290] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.521596 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:52:59.521760 4011 raft_consensus.cc:695] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:52:59.521782 3917 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.522117 4011 consensus_queue.cc:237] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.522313 4002 raft_consensus.cc:2802] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.522426 4002 raft_consensus.cc:491] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.522514 4002 raft_consensus.cc:3058] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.522511 3963 ts_tablet_manager.cc:1428] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250626 01:52:59.522927 3963 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.524670 3964 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.524834 3964 ts_tablet_manager.cc:1397] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.007s
I20250626 01:52:59.524945 3917 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.525199 3963 tablet_bootstrap.cc:654] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.525738 3964 raft_consensus.cc:357] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.526062 3964 raft_consensus.cc:383] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.526232 3964 raft_consensus.cc:738] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.526985 3964 consensus_queue.cc:260] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.527804 3964 ts_tablet_manager.cc:1428] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:52:59.528373 3964 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: Bootstrap starting.
I20250626 01:52:59.528185 3665 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 0 to 1, leader changed from <none> to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.531265 3964 tablet_bootstrap.cc:654] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.533244 3965 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: No bootstrap required, opened a new log
I20250626 01:52:59.533486 3965 ts_tablet_manager.cc:1397] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20250626 01:52:59.535310 3965 raft_consensus.cc:357] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.535616 3965 raft_consensus.cc:383] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.535681 4002 raft_consensus.cc:513] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.536386 4002 leader_election.cc:290] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.535769 3965 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.536741 3965 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.537202 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.537374 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.537375 3965 ts_tablet_manager.cc:1428] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250626 01:52:59.539252 3844 raft_consensus.cc:3058] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.538267 3963 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.540313 3963 ts_tablet_manager.cc:1397] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.015s
I20250626 01:52:59.541172 3964 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: No bootstrap required, opened a new log
I20250626 01:52:59.541402 3964 ts_tablet_manager.cc:1397] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20250626 01:52:59.541620 3963 raft_consensus.cc:357] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.541918 3963 raft_consensus.cc:383] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.542032 3963 raft_consensus.cc:738] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.542352 3963 consensus_queue.cc:260] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.542657 3844 raft_consensus.cc:2466] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.542976 3963 ts_tablet_manager.cc:1428] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:52:59.539240 3769 raft_consensus.cc:3058] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.543279 3884 leader_election.cc:304] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.543679 4002 raft_consensus.cc:2802] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.543839 4002 raft_consensus.cc:695] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.544126 4002 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.542366 3964 raft_consensus.cc:357] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.544948 3964 raft_consensus.cc:383] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.545042 3964 raft_consensus.cc:738] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.545305 3964 consensus_queue.cc:260] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.545372 3667 catalog_manager.cc:5582] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 0 to 1, leader changed from <none> to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.546160 3769 raft_consensus.cc:2466] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.546643 3963 tablet_bootstrap.cc:492] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: Bootstrap starting.
I20250626 01:52:59.547613 3964 ts_tablet_manager.cc:1428] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250626 01:52:59.548705 3963 tablet_bootstrap.cc:654] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: Neither blocks nor log segments found. Creating new log.
I20250626 01:52:59.552899 3970 raft_consensus.cc:491] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.553141 3970 raft_consensus.cc:513] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.553880 3970 leader_election.cc:290] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.554659 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.554947 3769 raft_consensus.cc:2466] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.554819 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.555585 3884 leader_election.cc:304] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.556120 3970 raft_consensus.cc:2802] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.556245 3970 raft_consensus.cc:491] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.556361 3970 raft_consensus.cc:3058] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.558800 3844 raft_consensus.cc:2466] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.559703 3963 tablet_bootstrap.cc:492] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: No bootstrap required, opened a new log
I20250626 01:52:59.559962 3963 ts_tablet_manager.cc:1397] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20250626 01:52:59.559935 3970 raft_consensus.cc:513] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.559711 3667 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:59.560671 3970 leader_election.cc:290] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.560904 3963 raft_consensus.cc:357] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.561159 3963 raft_consensus.cc:383] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:52:59.561223 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.561290 3963 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Initialized, Role: FOLLOWER
I20250626 01:52:59.561417 3769 raft_consensus.cc:3058] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.561581 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.561553 3963 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.561781 3844 raft_consensus.cc:3058] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.562175 3963 ts_tablet_manager.cc:1428] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:52:59.564241 3769 raft_consensus.cc:2466] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.564375 3844 raft_consensus.cc:2466] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.564761 3884 leader_election.cc:304] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.565055 3970 raft_consensus.cc:2802] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.565315 3970 raft_consensus.cc:695] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.565548 3970 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.568048 3667 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:59.574084 4011 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.574307 4011 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.575021 4011 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.575547 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925" is_pre_election: true
I20250626 01:52:59.575541 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.575807 3917 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.575816 3769 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 0.
I20250626 01:52:59.576323 3810 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.576639 4011 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.576761 4011 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.576856 4011 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.578910 4011 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.579530 4011 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:52:59.580092 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.580101 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:52:59.580339 3917 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.580348 3769 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.583266 3917 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.583263 3769 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 1.
I20250626 01:52:59.583868 3809 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:52:59.584239 4011 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.584426 4011 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:52:59.584730 4011 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.587554 3667 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 0 to 1, leader changed from <none> to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:52:59.604984 3970 raft_consensus.cc:491] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.605199 3970 raft_consensus.cc:513] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.605876 3970 leader_election.cc:290] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.606415 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.606509 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.606675 3769 raft_consensus.cc:2391] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 001b6ec9d9bc46c79e57a4915c44c925 in current term 1: Already voted for candidate 311ee4b8be9f47a8a3de56840e44425a in this term.
I20250626 01:52:59.607120 3884 leader_election.cc:304] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925; no voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:52:59.607991 3970 raft_consensus.cc:3058] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.609988 3970 raft_consensus.cc:2747] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 01:52:59.785682 3970 raft_consensus.cc:491] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.785950 3970 raft_consensus.cc:513] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.786645 3970 leader_election.cc:290] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.787281 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.787319 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.787475 3769 raft_consensus.cc:2466] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.787549 3844 raft_consensus.cc:2466] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 0.
I20250626 01:52:59.787936 3884 leader_election.cc:304] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.788233 3970 raft_consensus.cc:2802] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:52:59.788339 3970 raft_consensus.cc:491] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:52:59.788448 3970 raft_consensus.cc:3058] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.790721 3970 raft_consensus.cc:513] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.791378 3970 leader_election.cc:290] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.792052 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:52:59.792047 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.792359 3769 raft_consensus.cc:3058] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.792385 3844 raft_consensus.cc:3058] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.795765 3769 raft_consensus.cc:2466] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.795765 3844 raft_consensus.cc:2466] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 1.
I20250626 01:52:59.796473 3884 leader_election.cc:304] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:52:59.796761 3970 raft_consensus.cc:2802] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:52:59.796936 3970 raft_consensus.cc:695] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.797209 3970 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.800184 3667 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 0 to 1, leader changed from <none> to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:59.828949 4002 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.836956 4002 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.852696 3970 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:52:59.852901 3970 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:52:59.853674 3970 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.854002 3769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" is_pre_election: true
I20250626 01:52:59.854729 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "311ee4b8be9f47a8a3de56840e44425a" is_pre_election: true
I20250626 01:52:59.854998 3844 raft_consensus.cc:2391] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 001b6ec9d9bc46c79e57a4915c44c925 in current term 1: Already voted for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in this term.
I20250626 01:52:59.855381 3884 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925; no voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:52:59.855705 3970 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:52:59.858136 3970 raft_consensus.cc:2747] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250626 01:52:59.860221 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:52:59.871609 3917 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:52:59.871913 3917 raft_consensus.cc:604] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:52:59.872925 3917 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:52:59.873168 3917 raft_consensus.cc:604] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:52:59.874550 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 01:52:59.876143 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:52:59.879370 3970 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.882740 4002 raft_consensus.cc:991] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:52:59.882964 4002 raft_consensus.cc:1079] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:52:59.884317 3768 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:52:59.884618 3768 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:52:59.884831 3768 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:52:59.885098 3970 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.888520 3768 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.888957 3769 raft_consensus.cc:1238] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Rejecting Update request from peer 001b6ec9d9bc46c79e57a4915c44c925 for earlier term 1. Current term is 2. Ops: []
I20250626 01:52:59.890671 3917 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:52:59.891058 3917 raft_consensus.cc:3053] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:52:59.891110 3843 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:52:59.891216 3917 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:52:59.891727 3768 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 2 election: Requested vote from peers 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:52:59.891659 3917 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.892057 3917 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:52:59.897748 3969 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.906703 3969 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.910439 4011 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.914192 3969 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.921607 4010 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.924360 3917 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 2.
I20250626 01:52:59.924819 3734 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters: 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:52:59.925436 3969 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:52:59.925582 3969 raft_consensus.cc:695] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:52:59.925881 3969 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } }
I20250626 01:52:59.930424 3667 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 1 to 2, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } }
I20250626 01:52:59.935992 3969 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.954761 3970 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.972059 4002 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.979436 4010 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:52:59.985311 3898 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
I20250626 01:52:59.991729 3843 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:52:59.992612 3970 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.994659 3767 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:52:59.994998 3844 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:52:59.998510 4003 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:52:59.999271 4003 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:00.000614 3768 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:53:00.004139 3970 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:53:00.007565 3897 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.007922 4011 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:00.009550 4011 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:00.030441 4011 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:53:00.033133 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
W20250626 01:53:00.034585 3895 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.036722 3894 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.037442 3894 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
W20250626 01:53:00.038671 3893 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.043011 3841 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:53:00.044867 4002 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:00.050071 4060 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:53:00.051175 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.052845 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.053943 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.064306 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.110932 3895 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.116040 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.121191 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.123266 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
W20250626 01:53:00.127528 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.130143 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.131058 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.132242 4010 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250626 01:53:00.158852 3820 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
W20250626 01:53:00.160962 3819 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
W20250626 01:53:00.161947 3819 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
W20250626 01:53:00.165561 4063 log.cc:927] Time spent T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20250626 01:53:00.167987 3820 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
W20250626 01:53:00.187001 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.189990 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.190683 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.196413 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.208089 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.208631 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.210673 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.212921 3844 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:00.217872 3844 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:53:00.218895 4001 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 01:53:00.223680 3891 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.224288 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.226998 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
W20250626 01:53:00.231400 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.251788 4003 raft_consensus.cc:991] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:00.252004 4003 raft_consensus.cc:1079] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
W20250626 01:53:00.252169 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.253365 3767 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:53:00.253842 3767 raft_consensus.cc:491] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:00.254027 3767 raft_consensus.cc:3058] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:00.257962 3767 raft_consensus.cc:513] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:53:00.260375 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.262233 3767 leader_election.cc:290] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 2 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
W20250626 01:53:00.262692 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.262892 3844 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:00.263078 3844 raft_consensus.cc:3058] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
W20250626 01:53:00.263762 3895 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Service unavailable: leader transfer in progress
I20250626 01:53:00.263952 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "16e2c24dbd4f446caf8d09565cbac048" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:00.264248 3918 raft_consensus.cc:3053] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:00.264358 3918 raft_consensus.cc:738] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:00.264699 3918 consensus_queue.cc:260] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:00.265064 3918 raft_consensus.cc:3058] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:00.271056 4011 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:00.272665 3916 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250626 01:53:00.295553 3896 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
W20250626 01:53:00.296993 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
I20250626 01:53:00.276957 4090 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
W20250626 01:53:00.328734 3895 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
W20250626 01:53:00.336210 4072 log.cc:927] Time spent T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20250626 01:53:00.335963 3809 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:35233, user_credentials={real_user=slave}} blocked reactor thread for 111599us
W20250626 01:53:00.346525 3744 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41792: Illegal state: replica 7e5284db6ec74f4a992cbd71b1dfc53f is not leader of this config: current role FOLLOWER
W20250626 01:53:00.353158 3889 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:47184: Illegal state: replica 001b6ec9d9bc46c79e57a4915c44c925 is not leader of this config: current role FOLLOWER
I20250626 01:53:00.364363 3918 raft_consensus.cc:2466] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 2.
I20250626 01:53:00.364946 3734 leader_election.cc:304] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:00.373106 4090 raft_consensus.cc:2802] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election won for term 2
W20250626 01:53:00.373296 3744 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41792: Illegal state: replica 7e5284db6ec74f4a992cbd71b1dfc53f is not leader of this config: current role FOLLOWER
W20250626 01:53:00.374590 3747 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41792: Service unavailable: leader is not yet ready
W20250626 01:53:00.375871 4042 log.cc:927] Time spent T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20250626 01:53:00.378621 3823 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
I20250626 01:53:00.373275 4090 raft_consensus.cc:695] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
W20250626 01:53:00.388772 3745 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:41792: Service unavailable: leader is not yet ready
I20250626 01:53:00.380093 4090 consensus_queue.cc:237] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:00.400307 3844 raft_consensus.cc:2466] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 2.
I20250626 01:53:00.400714 3839 raft_consensus.cc:1238] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Rejecting Update request from peer 001b6ec9d9bc46c79e57a4915c44c925 for earlier term 1. Current term is 2. Ops: []
I20250626 01:53:00.426211 4071 mvcc.cc:204] Tried to move back new op lower bound from 7171697787626958848 to 7171697786582581248. Current Snapshot: MvccSnapshot[applied={T|T < 7171697787626958848}]
W20250626 01:53:00.430256 3823 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
W20250626 01:53:00.431253 3822 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
W20250626 01:53:00.434681 3821 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:33242: Illegal state: replica 311ee4b8be9f47a8a3de56840e44425a is not leader of this config: current role FOLLOWER
I20250626 01:53:00.466902 3659 catalog_manager.cc:5582] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 1 to 2, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:00.482875 3915 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:53:00.483469 4051 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:00.493813 3838 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:53:00.496629 4091 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 01:53:00.826757 3884 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:40297, user_credentials={real_user=slave}} blocked reactor thread for 54307.8us
I20250626 01:53:01.577854 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:01.585672 4117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:01.586125 4118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:01.588557 4120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:01.588627 413 server_base.cc:1048] running on GCE node
I20250626 01:53:01.588958 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:01.589018 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:01.589064 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902781589053 us; error 0 us; skew 500 ppm
I20250626 01:53:01.589214 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:01.590621 413 webserver.cc:469] Webserver started at http://127.0.103.68:46311/ using document root <none> and password file <none>
I20250626 01:53:01.590844 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:01.590915 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:01.591043 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:01.591665 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
format_stamp: "Formatted at 2025-06-26 01:53:01 on dist-test-slave-psm5"
I20250626 01:53:01.594523 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20250626 01:53:01.596287 4125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.596686 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:53:01.596798 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
format_stamp: "Formatted at 2025-06-26 01:53:01 on dist-test-slave-psm5"
I20250626 01:53:01.596925 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:01.615618 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:01.616302 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:01.617026 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:01.618135 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:01.618197 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.618271 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:01.618306 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.637955 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:44397
I20250626 01:53:01.637990 4187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:44397 every 8 connection(s)
I20250626 01:53:01.639815 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:01.643828 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:01.644500 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:01.645871 413 server_base.cc:1048] running on GCE node
W20250626 01:53:01.646018 4195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:01.646389 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:01.646447 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:01.646477 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902781646474 us; error 0 us; skew 500 ppm
I20250626 01:53:01.646615 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:01.648964 413 webserver.cc:469] Webserver started at http://127.0.103.69:43405/ using document root <none> and password file <none>
I20250626 01:53:01.649142 4188 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34867
I20250626 01:53:01.649232 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:01.649271 4188 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:01.649312 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:01.649441 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:01.649703 4188 heartbeater.cc:507] Master 127.0.103.126:34867 requested a full tablet report, sending...
I20250626 01:53:01.650095 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-4-root/instance:
uuid: "464c771a333746b6aab3fc7a84481415"
format_stamp: "Formatted at 2025-06-26 01:53:01 on dist-test-slave-psm5"
I20250626 01:53:01.650686 3660 ts_manager.cc:194] Registered new tserver with Master: fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:01.651968 3660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55150
I20250626 01:53:01.653518 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:53:01.655247 4200 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.655619 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:53:01.655711 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-4-root
uuid: "464c771a333746b6aab3fc7a84481415"
format_stamp: "Formatted at 2025-06-26 01:53:01 on dist-test-slave-psm5"
I20250626 01:53:01.666831 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:01.680109 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:01.680646 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:01.681167 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:01.681962 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:01.682017 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.682090 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:01.682132 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.698545 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:45827
I20250626 01:53:01.698603 4262 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:45827 every 8 connection(s)
I20250626 01:53:01.700253 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:01.703604 4266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:01.703822 4267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:01.706631 4263 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34867
I20250626 01:53:01.706769 4263 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:01.707172 4263 heartbeater.cc:507] Master 127.0.103.126:34867 requested a full tablet report, sending...
I20250626 01:53:01.708191 3660 ts_manager.cc:194] Registered new tserver with Master: 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:01.709281 3660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55156
W20250626 01:53:01.709496 4269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:01.709619 413 server_base.cc:1048] running on GCE node
I20250626 01:53:01.710206 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:01.710279 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:01.710314 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902781710310 us; error 0 us; skew 500 ppm
I20250626 01:53:01.710459 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:01.711848 413 webserver.cc:469] Webserver started at http://127.0.103.70:43793/ using document root <none> and password file <none>
I20250626 01:53:01.712080 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:01.712149 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:01.712239 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:01.712805 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-5-root/instance:
uuid: "9832bf960c714aff8c1dc4fc62ee4099"
format_stamp: "Formatted at 2025-06-26 01:53:01 on dist-test-slave-psm5"
I20250626 01:53:01.715209 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250626 01:53:01.716781 4274 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.717164 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:53:01.717259 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-5-root
uuid: "9832bf960c714aff8c1dc4fc62ee4099"
format_stamp: "Formatted at 2025-06-26 01:53:01 on dist-test-slave-psm5"
I20250626 01:53:01.717353 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:01.750918 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:01.751426 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:01.751971 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:01.752849 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:01.752914 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.752971 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:01.753001 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:01.769330 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:43493
I20250626 01:53:01.769379 4336 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:43493 every 8 connection(s)
I20250626 01:53:01.776300 4337 heartbeater.cc:344] Connected to a master server at 127.0.103.126:34867
I20250626 01:53:01.776448 4337 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:01.776829 4337 heartbeater.cc:507] Master 127.0.103.126:34867 requested a full tablet report, sending...
I20250626 01:53:01.777650 3660 ts_manager.cc:194] Registered new tserver with Master: 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493)
I20250626 01:53:01.778435 3660 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55160
I20250626 01:53:01.876586 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:01.879753 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:01.880024 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:01.883939 3837 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:01.886224 3918 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250626 01:53:01.886586 3765 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250626 01:53:01.886945 4088 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 01:53:01.887213 4094 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 01:53:01.889766 4087 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:01.890630 3765 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:01.892561 3918 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
W20250626 01:53:01.893842 3807 consensus_peers.cc:487] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fb20902f87437487c5f36d71de0b9b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:01.896051 3659 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index -1 to 26, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:01.905735 3765 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:01.908370 3918 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:01.908484 3837 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:01.908984 4091 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:01.909324 4051 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:01.911938 4090 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:01.912463 3837 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:01.912547 3918 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
W20250626 01:53:01.916599 3735 consensus_peers.cc:487] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f43f00d4508b48278aeea9fb37615694. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:01.917979 3667 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:01.920208 3837 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:01.922397 3765 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:01.923251 4092 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:01.925541 3918 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:01.926245 4092 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:01.926928 4088 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.928089 3765 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.932444 3767 consensus_queue.cc:237] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:01.934034 3667 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:01.935309 3916 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:01.937028 4115 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:01.937794 3837 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:01.937507 3916 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.941100 4089 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:01.942550 4089 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
W20250626 01:53:01.944245 3810 consensus_peers.cc:487] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71684e4b73c24f57b67e052454de9097. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:01.944437 3918 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
W20250626 01:53:01.947094 3732 consensus_peers.cc:487] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16e2c24dbd4f446caf8d09565cbac048. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:01.949878 3837 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:01.950520 3667 catalog_manager.cc:5582] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index -1 to 28, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:01.959024 3918 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:01.960975 3765 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:01.961061 3837 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:01.961555 3970 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:01.961830 4060 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:01.964499 4003 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.965456 3837 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.965485 3765 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.970762 3667 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:01.971200 3918 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
W20250626 01:53:01.974083 3884 consensus_peers.cc:487] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ce2bf1f3ecc44f7824389e2febfcb4f. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:01.974642 3765 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250626 01:53:01.975051 3837 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20250626 01:53:01.975788 3970 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 01:53:01.976079 4003 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 01:53:01.978693 3970 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:01.979482 3837 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:01.979660 3765 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
W20250626 01:53:01.982875 3884 consensus_peers.cc:487] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e80ca9ec9a27407fa4e5ae542fdad35d. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:01.985080 3659 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index -1 to 26, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:02.346370 4354 ts_tablet_manager.cc:927] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: Initiating tablet copy from peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:53:02.347072 4354 tablet_copy_client.cc:323] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:34305
I20250626 01:53:02.353579 3779 tablet_copy_service.cc:140] P 7e5284db6ec74f4a992cbd71b1dfc53f: Received BeginTabletCopySession request for tablet 16e2c24dbd4f446caf8d09565cbac048 from peer 9832bf960c714aff8c1dc4fc62ee4099 ({username='slave'} at 127.0.0.1:41818)
I20250626 01:53:02.353817 3779 tablet_copy_service.cc:161] P 7e5284db6ec74f4a992cbd71b1dfc53f: Beginning new tablet copy session on tablet 16e2c24dbd4f446caf8d09565cbac048 from peer 9832bf960c714aff8c1dc4fc62ee4099 at {username='slave'} at 127.0.0.1:41818: session id = 9832bf960c714aff8c1dc4fc62ee4099-16e2c24dbd4f446caf8d09565cbac048
I20250626 01:53:02.356549 3779 tablet_copy_source_session.cc:215] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:02.358125 4357 ts_tablet_manager.cc:927] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Initiating tablet copy from peer 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:02.358184 4354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16e2c24dbd4f446caf8d09565cbac048. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:02.358986 4357 tablet_copy_client.cc:323] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:40297
I20250626 01:53:02.366706 3854 tablet_copy_service.cc:140] P 311ee4b8be9f47a8a3de56840e44425a: Received BeginTabletCopySession request for tablet 96fb20902f87437487c5f36d71de0b9b from peer 9832bf960c714aff8c1dc4fc62ee4099 ({username='slave'} at 127.0.0.1:33254)
I20250626 01:53:02.367115 3854 tablet_copy_service.cc:161] P 311ee4b8be9f47a8a3de56840e44425a: Beginning new tablet copy session on tablet 96fb20902f87437487c5f36d71de0b9b from peer 9832bf960c714aff8c1dc4fc62ee4099 at {username='slave'} at 127.0.0.1:33254: session id = 9832bf960c714aff8c1dc4fc62ee4099-96fb20902f87437487c5f36d71de0b9b
I20250626 01:53:02.370996 3854 tablet_copy_source_session.cc:215] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:02.372473 4357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fb20902f87437487c5f36d71de0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:02.375931 4354 tablet_copy_client.cc:806] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:02.376281 4354 tablet_copy_client.cc:670] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:02.378846 4359 ts_tablet_manager.cc:927] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Initiating tablet copy from peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:02.379832 4359 tablet_copy_client.cc:323] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35233
I20250626 01:53:02.380184 4357 tablet_copy_client.cc:806] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:02.380472 4357 tablet_copy_client.cc:670] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:02.380774 4355 tablet_copy_client.cc:1025] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:02.382153 4358 tablet_copy_client.cc:1025] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:02.387217 3928 tablet_copy_service.cc:140] P 001b6ec9d9bc46c79e57a4915c44c925: Received BeginTabletCopySession request for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f from peer fb97ea56eeb04a3d9bf983411d630ee7 ({username='slave'} at 127.0.0.1:47200)
I20250626 01:53:02.387467 3928 tablet_copy_service.cc:161] P 001b6ec9d9bc46c79e57a4915c44c925: Beginning new tablet copy session on tablet 0ce2bf1f3ecc44f7824389e2febfcb4f from peer fb97ea56eeb04a3d9bf983411d630ee7 at {username='slave'} at 127.0.0.1:47200: session id = fb97ea56eeb04a3d9bf983411d630ee7-0ce2bf1f3ecc44f7824389e2febfcb4f
I20250626 01:53:02.389288 4362 ts_tablet_manager.cc:927] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Initiating tablet copy from peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:02.390410 4362 tablet_copy_client.cc:323] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35233
I20250626 01:53:02.390983 3928 tablet_copy_source_session.cc:215] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:02.392431 4359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:02.398476 3928 tablet_copy_service.cc:140] P 001b6ec9d9bc46c79e57a4915c44c925: Received BeginTabletCopySession request for tablet e80ca9ec9a27407fa4e5ae542fdad35d from peer 464c771a333746b6aab3fc7a84481415 ({username='slave'} at 127.0.0.1:47204)
I20250626 01:53:02.398818 3928 tablet_copy_service.cc:161] P 001b6ec9d9bc46c79e57a4915c44c925: Beginning new tablet copy session on tablet e80ca9ec9a27407fa4e5ae542fdad35d from peer 464c771a333746b6aab3fc7a84481415 at {username='slave'} at 127.0.0.1:47204: session id = 464c771a333746b6aab3fc7a84481415-e80ca9ec9a27407fa4e5ae542fdad35d
I20250626 01:53:02.399775 4359 tablet_copy_client.cc:806] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:02.400099 4359 tablet_copy_client.cc:670] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:02.401937 4360 tablet_copy_client.cc:1025] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:02.402177 3928 tablet_copy_source_session.cc:215] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:02.403437 4362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80ca9ec9a27407fa4e5ae542fdad35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:02.410360 4362 tablet_copy_client.cc:806] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:02.410676 4362 tablet_copy_client.cc:670] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:02.413272 4363 tablet_copy_client.cc:1025] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:02.433526 4365 ts_tablet_manager.cc:927] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: Initiating tablet copy from peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:53:02.434363 4365 tablet_copy_client.cc:323] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:34305
I20250626 01:53:02.440271 3779 tablet_copy_service.cc:140] P 7e5284db6ec74f4a992cbd71b1dfc53f: Received BeginTabletCopySession request for tablet f43f00d4508b48278aeea9fb37615694 from peer 464c771a333746b6aab3fc7a84481415 ({username='slave'} at 127.0.0.1:41826)
I20250626 01:53:02.440521 3779 tablet_copy_service.cc:161] P 7e5284db6ec74f4a992cbd71b1dfc53f: Beginning new tablet copy session on tablet f43f00d4508b48278aeea9fb37615694 from peer 464c771a333746b6aab3fc7a84481415 at {username='slave'} at 127.0.0.1:41826: session id = 464c771a333746b6aab3fc7a84481415-f43f00d4508b48278aeea9fb37615694
I20250626 01:53:02.443408 3779 tablet_copy_source_session.cc:215] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:02.444798 4365 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f43f00d4508b48278aeea9fb37615694. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:02.451555 4365 tablet_copy_client.cc:806] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:02.451814 4365 tablet_copy_client.cc:670] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:02.453485 4366 tablet_copy_client.cc:1025] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:02.514078 4367 ts_tablet_manager.cc:927] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: Initiating tablet copy from peer 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:02.514842 4367 tablet_copy_client.cc:323] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:40297
I20250626 01:53:02.521628 3854 tablet_copy_service.cc:140] P 311ee4b8be9f47a8a3de56840e44425a: Received BeginTabletCopySession request for tablet 71684e4b73c24f57b67e052454de9097 from peer fb97ea56eeb04a3d9bf983411d630ee7 ({username='slave'} at 127.0.0.1:33268)
I20250626 01:53:02.521780 3854 tablet_copy_service.cc:161] P 311ee4b8be9f47a8a3de56840e44425a: Beginning new tablet copy session on tablet 71684e4b73c24f57b67e052454de9097 from peer fb97ea56eeb04a3d9bf983411d630ee7 at {username='slave'} at 127.0.0.1:33268: session id = fb97ea56eeb04a3d9bf983411d630ee7-71684e4b73c24f57b67e052454de9097
I20250626 01:53:02.523788 3854 tablet_copy_source_session.cc:215] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:02.525049 4367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71684e4b73c24f57b67e052454de9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:02.531114 4367 tablet_copy_client.cc:806] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:02.531390 4367 tablet_copy_client.cc:670] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:02.532807 4368 tablet_copy_client.cc:1025] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:02.653383 4188 heartbeater.cc:499] Master 127.0.103.126:34867 was elected leader, sending a full tablet report...
I20250626 01:53:02.710927 4263 heartbeater.cc:499] Master 127.0.103.126:34867 was elected leader, sending a full tablet report...
I20250626 01:53:02.780217 4337 heartbeater.cc:499] Master 127.0.103.126:34867 was elected leader, sending a full tablet report...
I20250626 01:53:03.880441 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:03.884479 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:03.884773 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:04.382366 4354 tablet_copy_client.cc:538] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:04.384558 4357 tablet_copy_client.cc:538] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:04.385563 4354 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap starting.
I20250626 01:53:04.388376 4357 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap starting.
I20250626 01:53:04.403510 4359 tablet_copy_client.cc:538] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:04.407482 4359 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap starting.
I20250626 01:53:04.414986 4362 tablet_copy_client.cc:538] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:04.418921 4362 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Bootstrap starting.
I20250626 01:53:04.437565 4357 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:04.438030 4357 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap complete.
I20250626 01:53:04.438431 4357 ts_tablet_manager.cc:1397] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Time spent bootstrapping tablet: real 0.050s user 0.045s sys 0.000s
I20250626 01:53:04.439781 4357 raft_consensus.cc:357] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:04.440169 4357 raft_consensus.cc:738] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Initialized, Role: LEARNER
I20250626 01:53:04.440500 4357 consensus_queue.cc:260] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:04.441431 4357 ts_tablet_manager.cc:1428] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:53:04.444767 3854 tablet_copy_service.cc:342] P 311ee4b8be9f47a8a3de56840e44425a: Request end of tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-96fb20902f87437487c5f36d71de0b9b received from {username='slave'} at 127.0.0.1:33254
I20250626 01:53:04.444913 3854 tablet_copy_service.cc:434] P 311ee4b8be9f47a8a3de56840e44425a: ending tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-96fb20902f87437487c5f36d71de0b9b on tablet 96fb20902f87437487c5f36d71de0b9b with peer 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:04.455530 4365 tablet_copy_client.cc:538] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:04.465308 4365 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: Bootstrap starting.
I20250626 01:53:04.465355 4354 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=88 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:04.472277 4354 tablet_bootstrap.cc:492] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap complete.
I20250626 01:53:04.472576 4354 ts_tablet_manager.cc:1397] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: Time spent bootstrapping tablet: real 0.087s user 0.034s sys 0.016s
I20250626 01:53:04.473129 4362 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:04.473567 4362 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Bootstrap complete.
I20250626 01:53:04.473851 4362 ts_tablet_manager.cc:1397] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Time spent bootstrapping tablet: real 0.055s user 0.044s sys 0.001s
I20250626 01:53:04.474030 4354 raft_consensus.cc:357] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:04.474267 4354 raft_consensus.cc:738] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Initialized, Role: LEARNER
I20250626 01:53:04.474462 4354 consensus_queue.cc:260] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:04.474675 4362 raft_consensus.cc:357] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:04.474908 4362 raft_consensus.cc:738] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Initialized, Role: LEARNER
I20250626 01:53:04.475003 4354 ts_tablet_manager.cc:1428] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:53:04.475131 4362 consensus_queue.cc:260] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:04.478066 3779 tablet_copy_service.cc:342] P 7e5284db6ec74f4a992cbd71b1dfc53f: Request end of tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-16e2c24dbd4f446caf8d09565cbac048 received from {username='slave'} at 127.0.0.1:41818
I20250626 01:53:04.478215 3779 tablet_copy_service.cc:434] P 7e5284db6ec74f4a992cbd71b1dfc53f: ending tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-16e2c24dbd4f446caf8d09565cbac048 on tablet 16e2c24dbd4f446caf8d09565cbac048 with peer 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:04.483875 4362 ts_tablet_manager.cc:1428] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20250626 01:53:04.485818 3928 tablet_copy_service.cc:342] P 001b6ec9d9bc46c79e57a4915c44c925: Request end of tablet copy session 464c771a333746b6aab3fc7a84481415-e80ca9ec9a27407fa4e5ae542fdad35d received from {username='slave'} at 127.0.0.1:47204
I20250626 01:53:04.485976 3928 tablet_copy_service.cc:434] P 001b6ec9d9bc46c79e57a4915c44c925: ending tablet copy session 464c771a333746b6aab3fc7a84481415-e80ca9ec9a27407fa4e5ae542fdad35d on tablet e80ca9ec9a27407fa4e5ae542fdad35d with peer 464c771a333746b6aab3fc7a84481415
I20250626 01:53:04.505712 4359 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:04.506209 4359 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap complete.
I20250626 01:53:04.506525 4359 ts_tablet_manager.cc:1397] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Time spent bootstrapping tablet: real 0.099s user 0.053s sys 0.022s
I20250626 01:53:04.507292 4359 raft_consensus.cc:357] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:04.507509 4359 raft_consensus.cc:738] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Initialized, Role: LEARNER
I20250626 01:53:04.507807 4359 consensus_queue.cc:260] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:04.508615 4359 ts_tablet_manager.cc:1428] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250626 01:53:04.510931 3928 tablet_copy_service.cc:342] P 001b6ec9d9bc46c79e57a4915c44c925: Request end of tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-0ce2bf1f3ecc44f7824389e2febfcb4f received from {username='slave'} at 127.0.0.1:47200
I20250626 01:53:04.511060 3928 tablet_copy_service.cc:434] P 001b6ec9d9bc46c79e57a4915c44c925: ending tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-0ce2bf1f3ecc44f7824389e2febfcb4f on tablet 0ce2bf1f3ecc44f7824389e2febfcb4f with peer fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:04.522552 4365 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:04.522997 4365 tablet_bootstrap.cc:492] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: Bootstrap complete.
I20250626 01:53:04.523273 4365 ts_tablet_manager.cc:1397] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: Time spent bootstrapping tablet: real 0.058s user 0.038s sys 0.012s
I20250626 01:53:04.523917 4365 raft_consensus.cc:357] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:04.524091 4365 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Initialized, Role: LEARNER
I20250626 01:53:04.524268 4365 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:04.524713 4365 ts_tablet_manager.cc:1428] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20250626 01:53:04.525373 3779 tablet_copy_service.cc:342] P 7e5284db6ec74f4a992cbd71b1dfc53f: Request end of tablet copy session 464c771a333746b6aab3fc7a84481415-f43f00d4508b48278aeea9fb37615694 received from {username='slave'} at 127.0.0.1:41826
I20250626 01:53:04.525462 3779 tablet_copy_service.cc:434] P 7e5284db6ec74f4a992cbd71b1dfc53f: ending tablet copy session 464c771a333746b6aab3fc7a84481415-f43f00d4508b48278aeea9fb37615694 on tablet f43f00d4508b48278aeea9fb37615694 with peer 464c771a333746b6aab3fc7a84481415
I20250626 01:53:04.534260 4367 tablet_copy_client.cc:538] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:04.537581 4367 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap starting.
I20250626 01:53:04.572254 4367 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:04.572654 4367 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap complete.
I20250626 01:53:04.572924 4367 ts_tablet_manager.cc:1397] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: Time spent bootstrapping tablet: real 0.035s user 0.030s sys 0.005s
I20250626 01:53:04.573629 4367 raft_consensus.cc:357] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:04.573836 4367 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Initialized, Role: LEARNER
I20250626 01:53:04.574045 4367 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:04.574491 4367 ts_tablet_manager.cc:1428] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:53:04.575101 3854 tablet_copy_service.cc:342] P 311ee4b8be9f47a8a3de56840e44425a: Request end of tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-71684e4b73c24f57b67e052454de9097 received from {username='slave'} at 127.0.0.1:33268
I20250626 01:53:04.575217 3854 tablet_copy_service.cc:434] P 311ee4b8be9f47a8a3de56840e44425a: ending tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-71684e4b73c24f57b67e052454de9097 on tablet 71684e4b73c24f57b67e052454de9097 with peer fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:04.716831 4238 raft_consensus.cc:1215] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20250626 01:53:04.732412 4312 raft_consensus.cc:1215] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20250626 01:53:04.781193 4238 raft_consensus.cc:1215] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250626 01:53:04.792964 4312 raft_consensus.cc:1215] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20250626 01:53:04.928498 4163 raft_consensus.cc:1215] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20250626 01:53:05.098778 4163 raft_consensus.cc:1215] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20250626 01:53:05.212486 4385 raft_consensus.cc:1062] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: attempting to promote NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 to VOTER
I20250626 01:53:05.213436 4385 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:05.215797 4312 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:05.215869 3918 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:05.216027 3767 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:05.216449 4385 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:05.216723 4384 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:05.216976 4387 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:05.220330 4387 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.221014 3918 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.221009 4312 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.226662 3765 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.227972 3660 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 26 to 27, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.235782 3840 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:05.238314 3918 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.238314 4312 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.238834 4384 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.239125 4387 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.241413 4383 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.241986 3918 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.244796 4312 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.245352 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 96fb20902f87437487c5f36d71de0b9b with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:05.246485 3664 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.252120 3754 tablet_service.cc:1515] Processing DeleteTablet for tablet 96fb20902f87437487c5f36d71de0b9b with delete_type TABLET_DATA_TOMBSTONED (TS 7e5284db6ec74f4a992cbd71b1dfc53f not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:05.252883 4395 tablet_replica.cc:331] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:05.253118 4395 raft_consensus.cc:2241] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:05.253350 4395 raft_consensus.cc:2270] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:05.254535 4395 ts_tablet_manager.cc:1905] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:05.260170 4395 ts_tablet_manager.cc:1918] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250626 01:53:05.260308 4395 log.cc:1199] T 96fb20902f87437487c5f36d71de0b9b P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/wals/96fb20902f87437487c5f36d71de0b9b
I20250626 01:53:05.261022 3654 catalog_manager.cc:4928] TS 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): tablet 96fb20902f87437487c5f36d71de0b9b (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:05.271960 4397 raft_consensus.cc:1062] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: attempting to promote NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 to VOTER
I20250626 01:53:05.272795 4397 consensus_queue.cc:237] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:05.275631 3918 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:05.276116 3840 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:05.276563 4312 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:05.276626 4396 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.277213 4371 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.277964 4396 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.282174 4399 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.282804 4312 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.287120 3837 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.288645 3659 catalog_manager.cc:5582] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: config changed from index 28 to 29, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.290508 3918 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.294420 3970 raft_consensus.cc:1062] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: attempting to promote NON_VOTER 464c771a333746b6aab3fc7a84481415 to VOTER
I20250626 01:53:05.295482 3970 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:05.297669 3765 consensus_queue.cc:237] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:05.299839 4311 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:05.300863 4238 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:05.301440 4403 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:05.301909 3840 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:05.302410 4399 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:05.302644 4403 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:05.306142 3837 raft_consensus.cc:1273] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:05.305995 4403 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.307020 3840 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.311676 4399 consensus_queue.cc:1035] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:05.312489 3765 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:05.313459 3970 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:05.313292 4396 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.313681 4238 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.314105 4311 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.318023 3664 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 26 to 27, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.319226 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 16e2c24dbd4f446caf8d09565cbac048 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:05.320021 3667 catalog_manager.cc:5582] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.324600 3904 tablet_service.cc:1515] Processing DeleteTablet for tablet 16e2c24dbd4f446caf8d09565cbac048 with delete_type TABLET_DATA_TOMBSTONED (TS 001b6ec9d9bc46c79e57a4915c44c925 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:47144
I20250626 01:53:05.326395 3765 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.327152 3918 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:05.329912 3837 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.330600 4405 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.332674 3840 raft_consensus.cc:2953] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:05.336001 4237 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.336889 4408 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.001s
I20250626 01:53:05.341079 4405 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.341727 4411 tablet_replica.cc:331] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:05.342133 4411 raft_consensus.cc:2241] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:05.342386 4411 raft_consensus.cc:2270] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:05.343251 4411 ts_tablet_manager.cc:1905] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:05.343866 4397 raft_consensus.cc:1062] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: attempting to promote NON_VOTER 464c771a333746b6aab3fc7a84481415 to VOTER
I20250626 01:53:05.344713 4397 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:05.346235 3653 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e80ca9ec9a27407fa4e5ae542fdad35d with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:05.346880 4238 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.348325 3660 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.352253 3754 tablet_service.cc:1515] Processing DeleteTablet for tablet e80ca9ec9a27407fa4e5ae542fdad35d with delete_type TABLET_DATA_TOMBSTONED (TS 7e5284db6ec74f4a992cbd71b1dfc53f not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:05.352583 4395 tablet_replica.cc:331] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:05.352771 4395 raft_consensus.cc:2241] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:05.352952 4395 raft_consensus.cc:2270] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:05.354118 4395 ts_tablet_manager.cc:1905] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:05.354272 3837 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.356070 3918 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:05.356345 3840 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:05.356820 4396 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.357010 4238 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:05.357530 4396 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.357991 4396 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.365615 4395 ts_tablet_manager.cc:1918] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20250626 01:53:05.365801 4395 log.cc:1199] T e80ca9ec9a27407fa4e5ae542fdad35d P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/wals/e80ca9ec9a27407fa4e5ae542fdad35d
I20250626 01:53:05.366779 3654 catalog_manager.cc:4928] TS 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): tablet e80ca9ec9a27407fa4e5ae542fdad35d (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:05.369601 4411 ts_tablet_manager.cc:1918] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20250626 01:53:05.370545 4411 log.cc:1199] T 16e2c24dbd4f446caf8d09565cbac048 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/wals/16e2c24dbd4f446caf8d09565cbac048
I20250626 01:53:05.371701 3653 catalog_manager.cc:4928] TS 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): tablet 16e2c24dbd4f446caf8d09565cbac048 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:05.373708 4397 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.377128 3840 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.381351 4238 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.381955 3664 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.387671 3918 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.391988 3767 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:05.393952 4238 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:05.394431 3918 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:05.394533 4399 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:05.395035 4396 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:05.397029 4396 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.397692 4238 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.401075 3918 raft_consensus.cc:2953] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.402361 3660 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 reported cstate change: config changed from index 29 to 30, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:05.404098 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f43f00d4508b48278aeea9fb37615694 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:05.407588 3826 tablet_service.cc:1515] Processing DeleteTablet for tablet f43f00d4508b48278aeea9fb37615694 with delete_type TABLET_DATA_TOMBSTONED (TS 311ee4b8be9f47a8a3de56840e44425a not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:05.408459 4416 tablet_replica.cc:331] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:05.408655 4416 raft_consensus.cc:2241] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:05.408915 4416 raft_consensus.cc:2270] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:05.409765 4416 ts_tablet_manager.cc:1905] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:05.414791 4416 ts_tablet_manager.cc:1918] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20250626 01:53:05.414930 4416 log.cc:1199] T f43f00d4508b48278aeea9fb37615694 P 311ee4b8be9f47a8a3de56840e44425a: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/wals/f43f00d4508b48278aeea9fb37615694
I20250626 01:53:05.415607 3654 catalog_manager.cc:4928] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): tablet f43f00d4508b48278aeea9fb37615694 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:05.467708 4403 raft_consensus.cc:1062] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: attempting to promote NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 to VOTER
I20250626 01:53:05.468528 4403 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.470755 4163 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.471091 3840 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.471269 3767 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.471402 4405 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.471853 3970 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.472177 4403 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.479775 3970 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.480384 3767 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.481611 3837 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.485462 4163 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.486984 3659 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.494833 3916 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.496932 4163 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 01:53:05.497581 3970 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.499161 3767 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 01:53:05.499797 3970 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.500530 4405 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.504534 3653 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:05.505333 4163 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.509737 3664 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 28 to 29, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.512532 3767 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.514947 3826 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f with delete_type TABLET_DATA_TOMBSTONED (TS 311ee4b8be9f47a8a3de56840e44425a not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:05.515350 4416 tablet_replica.cc:331] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:05.515570 4416 raft_consensus.cc:2241] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:05.515854 4416 raft_consensus.cc:2270] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:05.516866 4416 ts_tablet_manager.cc:1905] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:05.522651 4416 ts_tablet_manager.cc:1918] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20250626 01:53:05.522900 4416 log.cc:1199] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 311ee4b8be9f47a8a3de56840e44425a: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/wals/0ce2bf1f3ecc44f7824389e2febfcb4f
I20250626 01:53:05.524869 4385 raft_consensus.cc:1062] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: attempting to promote NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 to VOTER
I20250626 01:53:05.525610 4385 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.527683 4163 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.527876 3918 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.528187 3767 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20250626 01:53:05.528198 4383 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.528652 4384 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.528858 4383 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:05.533210 4387 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.534212 3767 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.537295 3654 catalog_manager.cc:4928] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): tablet 0ce2bf1f3ecc44f7824389e2febfcb4f (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:05.537616 4163 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.538586 3664 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.539983 3918 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.544981 3837 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.546715 4163 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 01:53:05.547318 4387 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.547542 3767 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20250626 01:53:05.548096 4384 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:05.550140 4384 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.552861 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 71684e4b73c24f57b67e052454de9097 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:05.553790 4163 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.553725 3767 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:05.554461 3664 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 28 to 29, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:05.558943 3904 tablet_service.cc:1515] Processing DeleteTablet for tablet 71684e4b73c24f57b67e052454de9097 with delete_type TABLET_DATA_TOMBSTONED (TS 001b6ec9d9bc46c79e57a4915c44c925 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:47144
I20250626 01:53:05.560040 4411 tablet_replica.cc:331] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:05.560273 4411 raft_consensus.cc:2241] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:05.560542 4411 raft_consensus.cc:2270] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:05.561348 4411 ts_tablet_manager.cc:1905] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:05.566795 4411 ts_tablet_manager.cc:1918] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20250626 01:53:05.566912 4411 log.cc:1199] T 71684e4b73c24f57b67e052454de9097 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/wals/71684e4b73c24f57b67e052454de9097
I20250626 01:53:05.567516 3653 catalog_manager.cc:4928] TS 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): tablet 71684e4b73c24f57b67e052454de9097 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:05.885305 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:05.888126 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:05.888345 3918 raft_consensus.cc:604] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Received request to transfer leadership to TS fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:05.898561 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:05.898811 3840 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:05.899971 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "978023fb60024f26a213be77e43d8140"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:05.900132 3840 raft_consensus.cc:604] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:05.906396 3767 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:05.906635 3767 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:05.908221 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "464c771a333746b6aab3fc7a84481415"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:05.908411 3918 raft_consensus.cc:604] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Received request to transfer leadership to TS 464c771a333746b6aab3fc7a84481415
I20250626 01:53:05.909617 3767 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:05.909792 3767 raft_consensus.cc:604] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:05.910167 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20250626 01:53:05.910379 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:05.954016 4408 raft_consensus.cc:991] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:05.954211 4408 raft_consensus.cc:1079] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Signalling peer fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:05.954882 4163 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f"
dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:54818
I20250626 01:53:05.955072 4163 raft_consensus.cc:491] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:05.955154 4163 raft_consensus.cc:3058] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:05.957333 4163 raft_consensus.cc:513] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.958164 4163 leader_election.cc:290] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 2 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:05.958804 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:05.959184 3918 raft_consensus.cc:3053] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:05.959272 3918 raft_consensus.cc:738] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:05.959589 3918 consensus_queue.cc:260] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.959949 3918 raft_consensus.cc:3058] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:05.962801 3918 raft_consensus.cc:2466] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 2.
I20250626 01:53:05.963451 4128 leader_election.cc:304] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, fb97ea56eeb04a3d9bf983411d630ee7; no voters:
I20250626 01:53:05.963716 4381 raft_consensus.cc:2802] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:05.964188 4381 raft_consensus.cc:695] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Becoming Leader. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Running, Role: LEADER
I20250626 01:53:05.964288 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0ce2bf1f3ecc44f7824389e2febfcb4f" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:05.964488 3767 raft_consensus.cc:3058] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:05.964500 4381 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:05.966904 3767 raft_consensus.cc:2466] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 2.
I20250626 01:53:05.967262 3660 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: term changed from 1 to 2, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68). New cstate: current_term: 2 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:06.183790 4399 raft_consensus.cc:991] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:06.184003 4399 raft_consensus.cc:1079] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:06.184729 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33226
I20250626 01:53:06.184926 3840 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:06.185038 3840 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.187307 3840 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.188122 3840 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 2 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:06.188939 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:06.189123 3918 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.189137 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:06.189373 3767 raft_consensus.cc:3053] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:06.189448 3767 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:06.189750 3767 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.190116 3767 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.192272 4431 raft_consensus.cc:991] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:06.192272 3767 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 2.
I20250626 01:53:06.192472 4431 raft_consensus.cc:1079] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:06.192927 3810 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:06.192971 3918 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 2.
I20250626 01:53:06.193277 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "978023fb60024f26a213be77e43d8140"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:06.193507 3916 raft_consensus.cc:491] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:06.193575 4431 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:06.193603 3916 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.193696 4431 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:06.193938 4431 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.196185 3916 raft_consensus.cc:513] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.196828 3916 leader_election.cc:290] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 2 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:06.196920 3660 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 1 to 2, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 2 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:06.197124 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:06.197137 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:06.197352 3767 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.197361 3840 raft_consensus.cc:3053] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:06.197446 3840 raft_consensus.cc:738] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:06.197659 3840 consensus_queue.cc:260] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.197976 3840 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.200021 3767 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 2.
I20250626 01:53:06.200474 3884 leader_election.cc:304] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:06.200716 4408 raft_consensus.cc:2802] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:06.200858 4408 raft_consensus.cc:695] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:06.201092 4408 consensus_queue.cc:237] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 25, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.201998 3840 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 2.
I20250626 01:53:06.203713 3660 catalog_manager.cc:5582] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 1 to 2, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:06.327245 4403 raft_consensus.cc:991] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 464c771a333746b6aab3fc7a84481415 to start an election
I20250626 01:53:06.327409 4403 raft_consensus.cc:1079] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Signalling peer 464c771a333746b6aab3fc7a84481415 to start an election
I20250626 01:53:06.328064 4238 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d"
dest_uuid: "464c771a333746b6aab3fc7a84481415"
from {username='slave'} at 127.0.0.1:47250
I20250626 01:53:06.328239 4238 raft_consensus.cc:491] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:06.328331 4238 raft_consensus.cc:3058] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.329322 4384 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:06.329443 4384 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:06.330041 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:06.330219 3916 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:06.330307 3916 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.331080 4238 raft_consensus.cc:513] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:06.331952 4238 leader_election.cc:290] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [CANDIDATE]: Term 2 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:06.332489 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d" candidate_uuid: "464c771a333746b6aab3fc7a84481415" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:06.332656 3918 raft_consensus.cc:3053] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:06.332711 3918 raft_consensus.cc:738] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:06.332958 3918 consensus_queue.cc:260] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:06.333135 3916 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.333312 3918 raft_consensus.cc:3058] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.333703 3916 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 2 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:06.334213 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:06.334371 3767 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.334306 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:06.334481 3840 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:06.334535 3840 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:06.334898 3840 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.335160 3840 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.337459 3840 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 2.
I20250626 01:53:06.337587 3767 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 2.
I20250626 01:53:06.337996 3884 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:53:06.338258 4403 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:06.338389 4403 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:06.338640 4403 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:06.339089 3918 raft_consensus.cc:2466] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464c771a333746b6aab3fc7a84481415 in term 2.
I20250626 01:53:06.339571 4203 leader_election.cc:304] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 464c771a333746b6aab3fc7a84481415; no voters:
I20250626 01:53:06.340293 4436 raft_consensus.cc:2802] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:06.342281 3660 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 1 to 2, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:06.342988 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e80ca9ec9a27407fa4e5ae542fdad35d" candidate_uuid: "464c771a333746b6aab3fc7a84481415" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:06.343163 3840 raft_consensus.cc:3058] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:06.343652 4436 raft_consensus.cc:695] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 LEADER]: Becoming Leader. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Running, Role: LEADER
I20250626 01:53:06.344048 4436 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:06.346045 3840 raft_consensus.cc:2466] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464c771a333746b6aab3fc7a84481415 in term 2.
I20250626 01:53:06.346980 3660 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 reported cstate change: term changed from 1 to 2, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 464c771a333746b6aab3fc7a84481415 (127.0.103.69). New cstate: current_term: 2 leader_uuid: "464c771a333746b6aab3fc7a84481415" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:06.366863 4430 raft_consensus.cc:991] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:06.367044 4430 raft_consensus.cc:1079] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:06.367846 3918 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47160
I20250626 01:53:06.368088 3918 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:06.368187 3918 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:06.370085 3918 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:06.370697 3918 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 3 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:06.371351 4238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:06.371352 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:06.371600 3767 raft_consensus.cc:3053] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:06.371600 4238 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:06.371713 3767 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:06.372062 3767 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:06.372447 3767 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:06.375032 4238 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 3.
I20250626 01:53:06.375440 3767 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 3.
I20250626 01:53:06.375568 3884 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 464c771a333746b6aab3fc7a84481415; no voters:
I20250626 01:53:06.375856 4403 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:06.376016 4403 raft_consensus.cc:695] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:06.376256 4403 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:06.378475 3660 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 2 to 3, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 3 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:06.461153 3767 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:06.461838 4381 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:06.468835 3918 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:06.469624 4429 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:06.562517 3918 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:06.565069 3767 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20250626 01:53:06.565162 3840 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20250626 01:53:06.565330 3884 consensus_peers.cc:487] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eec015ad4dee46efb54559801ff9896b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:06.565696 4408 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:06.566025 4443 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:06.569475 4443 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:06.570230 3767 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:06.570263 3840 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:06.576108 3660 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index -1 to 27, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
W20250626 01:53:06.576894 3715 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20250626 01:53:06.581064 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 978023fb60024f26a213be77e43d8140, TS 311ee4b8be9f47a8a3de56840e44425a -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:53:06.581167 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 978023fb60024f26a213be77e43d8140, TS 311ee4b8be9f47a8a3de56840e44425a -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:53:06.589143 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0ce2bf1f3ecc44f7824389e2febfcb4f, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
W20250626 01:53:06.589250 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0ce2bf1f3ecc44f7824389e2febfcb4f, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
W20250626 01:53:06.598135 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet e80ca9ec9a27407fa4e5ae542fdad35d, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
W20250626 01:53:06.598273 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet e80ca9ec9a27407fa4e5ae542fdad35d, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
W20250626 01:53:06.601109 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 34f950fcb4ee4d5aad0d3f06ed2081af, TS 311ee4b8be9f47a8a3de56840e44425a -> TS fb97ea56eeb04a3d9bf983411d630ee7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:53:06.601210 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 34f950fcb4ee4d5aad0d3f06ed2081af, TS 311ee4b8be9f47a8a3de56840e44425a -> TS fb97ea56eeb04a3d9bf983411d630ee7 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:53:06.603641 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 71684e4b73c24f57b67e052454de9097, TS 7e5284db6ec74f4a992cbd71b1dfc53f -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
W20250626 01:53:06.603740 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 71684e4b73c24f57b67e052454de9097, TS 7e5284db6ec74f4a992cbd71b1dfc53f -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:06.658672 3767 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250626 01:53:06.659273 4408 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 01:53:06.663906 3840 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 2 index: 26. (index mismatch)
I20250626 01:53:06.664597 4408 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20250626 01:53:06.679427 3918 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:06.680065 4384 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:06.684501 3767 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:06.685431 4384 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:06.759478 3767 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:06.760154 4443 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:06.764253 3840 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:06.765651 4450 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:06.806594 3767 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 01:53:06.807119 4450 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:06.811183 4238 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 01:53:06.811743 4450 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:06.886924 3840 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:06.887465 4436 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:06.893354 3918 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:06.894301 4462 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:07.065456 4465 ts_tablet_manager.cc:927] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Initiating tablet copy from peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:07.066074 4465 tablet_copy_client.cc:323] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35233
I20250626 01:53:07.066598 3928 tablet_copy_service.cc:140] P 001b6ec9d9bc46c79e57a4915c44c925: Received BeginTabletCopySession request for tablet eec015ad4dee46efb54559801ff9896b from peer fb97ea56eeb04a3d9bf983411d630ee7 ({username='slave'} at 127.0.0.1:47200)
I20250626 01:53:07.066733 3928 tablet_copy_service.cc:161] P 001b6ec9d9bc46c79e57a4915c44c925: Beginning new tablet copy session on tablet eec015ad4dee46efb54559801ff9896b from peer fb97ea56eeb04a3d9bf983411d630ee7 at {username='slave'} at 127.0.0.1:47200: session id = fb97ea56eeb04a3d9bf983411d630ee7-eec015ad4dee46efb54559801ff9896b
I20250626 01:53:07.069445 3928 tablet_copy_source_session.cc:215] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:07.070583 4465 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec015ad4dee46efb54559801ff9896b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:07.076447 4465 tablet_copy_client.cc:806] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:07.076748 4465 tablet_copy_client.cc:670] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:07.078454 4466 tablet_copy_client.cc:1025] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:07.910777 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:07.914109 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:07.914369 3918 raft_consensus.cc:604] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:53:07.915489 3837 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:07.915683 3837 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:07.917196 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:07.917471 3916 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:07.918843 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:07.919147 3918 raft_consensus.cc:604] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:53:07.919631 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 01:53:07.919921 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:07.930364 4408 raft_consensus.cc:991] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:07.930548 4450 raft_consensus.cc:1079] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:07.931195 3767 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:53:07.931375 3767 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:07.931475 3767 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:07.933837 3767 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:07.934554 3767 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 4 election: Requested vote from peers 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:07.934921 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:07.935051 4238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:07.935138 3916 raft_consensus.cc:3053] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 01:53:07.935216 3916 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:07.935215 4238 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:07.935628 3916 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:07.936082 3916 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:07.938527 4238 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 4.
I20250626 01:53:07.938927 3916 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 4.
I20250626 01:53:07.939106 3735 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464c771a333746b6aab3fc7a84481415, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:07.939458 4472 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:53:07.939666 4472 raft_consensus.cc:695] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:07.940059 4472 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:07.943531 3664 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 3 to 4, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 4 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:08.037672 4408 raft_consensus.cc:991] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:08.037873 4450 raft_consensus.cc:1079] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:08.038450 3767 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:53:08.038645 3767 raft_consensus.cc:491] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:08.038754 3767 raft_consensus.cc:3058] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:08.041134 3767 raft_consensus.cc:513] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:08.041759 3767 leader_election.cc:290] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 2 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:08.042029 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:08.042232 3837 raft_consensus.cc:3058] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:08.042326 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "20ad5196a1fa4032afaa62ccab7e3fd1" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:08.042495 3918 raft_consensus.cc:3053] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:08.042557 3918 raft_consensus.cc:738] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:08.042806 3918 consensus_queue.cc:260] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:08.043154 3918 raft_consensus.cc:3058] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:08.044808 3837 raft_consensus.cc:2466] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 2.
I20250626 01:53:08.045248 3735 leader_election.cc:304] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:08.045538 4472 raft_consensus.cc:2802] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:08.045662 4472 raft_consensus.cc:695] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:08.045730 3918 raft_consensus.cc:2466] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 2.
I20250626 01:53:08.045878 4472 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:08.049294 3664 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 1 to 2, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:08.244421 4454 raft_consensus.cc:991] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:08.244622 4454 raft_consensus.cc:1079] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:08.245203 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:08.245361 3916 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:08.245483 3916 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:08.247941 3916 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:08.248616 3916 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 3 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:08.249142 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:08.249142 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:08.249359 3767 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:08.249378 3837 raft_consensus.cc:3053] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:08.249447 3837 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:08.249682 3837 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:08.249944 3837 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:08.252128 3767 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 3.
I20250626 01:53:08.252501 3837 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 3.
I20250626 01:53:08.252660 3884 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:08.252996 4480 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:08.253139 4480 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:08.254874 4480 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:08.270282 3664 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 2 to 3, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 3 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:08.298468 4408 raft_consensus.cc:991] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:08.298683 4408 raft_consensus.cc:1079] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:08.299225 3837 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33212
I20250626 01:53:08.299438 3837 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:08.299549 3837 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:08.302362 3837 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:08.303194 3837 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 2 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:08.303475 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:08.303659 3767 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:08.303614 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:08.303817 3918 raft_consensus.cc:3053] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:08.303916 3918 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:08.304172 3918 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:08.304543 3918 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:08.305787 3767 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 2.
I20250626 01:53:08.306216 3810 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:08.306510 4454 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:08.306675 4454 raft_consensus.cc:695] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:08.306975 4454 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:08.307168 3918 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 2.
I20250626 01:53:08.310501 3664 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 1 to 2, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 2 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:08.344619 4238 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 4 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 01:53:08.345155 4472 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:08.357215 3916 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 01:53:08.357937 4472 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:08.506153 3916 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:08.506692 4472 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:08.510831 3840 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:08.511399 4430 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:08.664893 3767 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:08.665452 4408 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:08.675484 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:08.676060 4408 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20250626 01:53:08.807801 3810 consensus_peers.cc:487] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:08.871014 3765 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:08.871997 4454 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:08.880518 3916 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:08.881266 4454 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:09.079398 4465 tablet_copy_client.cc:538] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:09.082707 4465 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap starting.
I20250626 01:53:09.128180 4465 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:09.128616 4465 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap complete.
I20250626 01:53:09.128907 4465 ts_tablet_manager.cc:1397] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Time spent bootstrapping tablet: real 0.046s user 0.046s sys 0.000s
I20250626 01:53:09.129703 4465 raft_consensus.cc:357] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:09.129937 4465 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Initialized, Role: LEARNER
I20250626 01:53:09.130167 4465 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:09.130780 4465 ts_tablet_manager.cc:1428] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:53:09.131394 3928 tablet_copy_service.cc:342] P 001b6ec9d9bc46c79e57a4915c44c925: Request end of tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-eec015ad4dee46efb54559801ff9896b received from {username='slave'} at 127.0.0.1:47200
I20250626 01:53:09.131497 3928 tablet_copy_service.cc:434] P 001b6ec9d9bc46c79e57a4915c44c925: ending tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-eec015ad4dee46efb54559801ff9896b on tablet eec015ad4dee46efb54559801ff9896b with peer fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:09.366609 4163 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 LEARNER]: Advancing to term 2
I20250626 01:53:09.789556 4506 raft_consensus.cc:1062] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: attempting to promote NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 to VOTER
I20250626 01:53:09.790254 4506 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 27, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:09.792330 3765 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:09.792337 4163 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:09.792337 3916 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:09.792913 4506 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:09.793183 4507 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:09.793491 4497 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:09.796669 4506 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 27 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.797286 3765 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.797516 4163 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.802083 3916 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 27 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.802580 3664 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 27 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.812207 3837 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:09.814723 4163 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:09.814755 3916 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:09.815276 4497 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:09.815585 4507 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:09.817996 4505 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.818580 4163 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.818709 3916 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.823050 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet eec015ad4dee46efb54559801ff9896b with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:09.823248 3667 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 29 to 30, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:09.827256 3754 tablet_service.cc:1515] Processing DeleteTablet for tablet eec015ad4dee46efb54559801ff9896b with delete_type TABLET_DATA_TOMBSTONED (TS 7e5284db6ec74f4a992cbd71b1dfc53f not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:09.827919 4512 tablet_replica.cc:331] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:09.828133 4512 raft_consensus.cc:2241] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:09.828420 4512 raft_consensus.cc:2270] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:09.829273 4512 ts_tablet_manager.cc:1905] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:09.833793 4512 ts_tablet_manager.cc:1918] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20250626 01:53:09.833914 4512 log.cc:1199] T eec015ad4dee46efb54559801ff9896b P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/wals/eec015ad4dee46efb54559801ff9896b
I20250626 01:53:09.834686 3654 catalog_manager.cc:4928] TS 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): tablet eec015ad4dee46efb54559801ff9896b (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:09.920490 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:09.923411 3765 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:09.923640 3765 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:09.924860 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:09.925042 3916 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:09.926016 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "9832bf960c714aff8c1dc4fc62ee4099"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:09.926158 3840 raft_consensus.cc:604] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Received request to transfer leadership to TS 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:09.927095 3765 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:09.927243 3765 raft_consensus.cc:604] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:09.927624 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20250626 01:53:09.927862 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:10.044281 4513 raft_consensus.cc:991] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:10.044438 4513 raft_consensus.cc:1079] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:10.045150 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47160
I20250626 01:53:10.045326 3916 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:10.045399 3916 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:10.047231 3916 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:10.047806 3916 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 5 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:10.048513 4238 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:10.048513 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:10.048720 4238 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:10.048720 3765 raft_consensus.cc:3053] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Stepping down as leader of term 4
I20250626 01:53:10.048816 3765 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:10.049121 3765 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:10.049487 3765 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:10.051805 4238 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 5.
I20250626 01:53:10.052067 3765 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 5.
I20250626 01:53:10.052431 3884 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:10.052687 4502 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 01:53:10.053092 4502 raft_consensus.cc:695] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:10.053364 4502 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:10.056294 3660 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 4 to 5, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:10.232518 4514 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:10.232695 4514 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:10.233374 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33212
I20250626 01:53:10.233551 3840 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:10.233650 3840 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:10.235613 3840 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.236256 3840 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 3 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:10.236784 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:10.236788 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:10.236959 3765 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:10.236958 3916 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:10.237061 3916 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:10.237295 3916 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.237561 3916 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:10.239109 3765 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 3.
I20250626 01:53:10.239377 3916 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 3.
I20250626 01:53:10.239559 3810 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:10.239918 4505 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:10.240056 4505 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:10.240281 4505 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.242862 3660 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 2 to 3, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:10.320186 4506 raft_consensus.cc:991] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 9832bf960c714aff8c1dc4fc62ee4099 to start an election
I20250626 01:53:10.320395 4505 raft_consensus.cc:1079] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Signalling peer 9832bf960c714aff8c1dc4fc62ee4099 to start an election
I20250626 01:53:10.321108 4311 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b"
dest_uuid: "9832bf960c714aff8c1dc4fc62ee4099"
from {username='slave'} at 127.0.0.1:60412
I20250626 01:53:10.321314 4311 raft_consensus.cc:491] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:10.321399 4311 raft_consensus.cc:3058] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:10.323484 4311 raft_consensus.cc:513] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:10.324343 4311 leader_election.cc:290] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [CANDIDATE]: Term 2 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:10.324952 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "9832bf960c714aff8c1dc4fc62ee4099" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:10.325198 3840 raft_consensus.cc:3053] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:10.325294 3840 raft_consensus.cc:738] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:10.325568 3840 consensus_queue.cc:260] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:10.325834 3840 raft_consensus.cc:3058] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:10.328212 3840 raft_consensus.cc:2466] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9832bf960c714aff8c1dc4fc62ee4099 in term 2.
I20250626 01:53:10.328682 4278 leader_election.cc:304] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 9832bf960c714aff8c1dc4fc62ee4099; no voters:
I20250626 01:53:10.329353 4519 raft_consensus.cc:2802] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:10.329859 4519 raft_consensus.cc:695] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEADER]: Becoming Leader. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Running, Role: LEADER
I20250626 01:53:10.330166 4519 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:10.330786 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96fb20902f87437487c5f36d71de0b9b" candidate_uuid: "9832bf960c714aff8c1dc4fc62ee4099" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:10.330966 3916 raft_consensus.cc:3058] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:10.332854 3660 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: term changed from 1 to 2, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70). New cstate: current_term: 2 leader_uuid: "9832bf960c714aff8c1dc4fc62ee4099" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:10.333520 3916 raft_consensus.cc:2466] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9832bf960c714aff8c1dc4fc62ee4099 in term 2.
I20250626 01:53:10.364620 4495 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:10.364817 4495 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:10.365427 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47160
I20250626 01:53:10.365609 3916 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:10.365712 3916 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:10.368122 3916 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.368826 3916 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 2 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:10.369336 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:10.369338 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:10.369508 3765 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:10.369575 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:10.369575 3765 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:10.369922 3765 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.370210 3765 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:10.371775 3840 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 2.
I20250626 01:53:10.372269 3884 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:53:10.372285 3765 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 2.
I20250626 01:53:10.372761 4514 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:10.372871 4514 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:10.373114 4514 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.375864 3660 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 1 to 2, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:10.539335 4238 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250626 01:53:10.539842 4514 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:10.543663 3765 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250626 01:53:10.544198 4502 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:10.627837 3916 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:10.628505 4505 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:10.632376 3765 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:10.633018 4505 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:10.829993 3765 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:10.831171 3916 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:10.831665 4519 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:10.832836 3840 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20250626 01:53:10.833005 3918 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20250626 01:53:10.833031 3735 consensus_peers.cc:487] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20ad5196a1fa4032afaa62ccab7e3fd1. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:10.833395 4533 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:10.833781 4513 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:10.845443 4533 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:10.847942 3916 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:10.850831 3837 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:10.851734 4519 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:10.852586 3837 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
W20250626 01:53:10.857829 3715 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250626 01:53:10.857906 3660 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index -1 to 28, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:10.863073 3765 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
W20250626 01:53:10.864359 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 346b250db608442d84f96d2c2e603775, TS 311ee4b8be9f47a8a3de56840e44425a -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
W20250626 01:53:10.864478 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 346b250db608442d84f96d2c2e603775, TS 311ee4b8be9f47a8a3de56840e44425a -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:10.864768 4502 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20250626 01:53:10.868213 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f43f00d4508b48278aeea9fb37615694, TS 7e5284db6ec74f4a992cbd71b1dfc53f -> TS fb97ea56eeb04a3d9bf983411d630ee7 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
W20250626 01:53:10.868327 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f43f00d4508b48278aeea9fb37615694, TS 7e5284db6ec74f4a992cbd71b1dfc53f -> TS fb97ea56eeb04a3d9bf983411d630ee7 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
W20250626 01:53:10.871619 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 16e2c24dbd4f446caf8d09565cbac048, TS 311ee4b8be9f47a8a3de56840e44425a -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
W20250626 01:53:10.871724 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 16e2c24dbd4f446caf8d09565cbac048, TS 311ee4b8be9f47a8a3de56840e44425a -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:10.874411 3840 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20250626 01:53:10.875002 4502 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
W20250626 01:53:10.877866 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet eec015ad4dee46efb54559801ff9896b, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
W20250626 01:53:10.878028 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet eec015ad4dee46efb54559801ff9896b, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 9832bf960c714aff8c1dc4fc62ee4099 move failed, destination replica disappeared from tablet's Raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
W20250626 01:53:10.881268 3715 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0ce2bf1f3ecc44f7824389e2febfcb4f, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
W20250626 01:53:10.881367 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0ce2bf1f3ecc44f7824389e2febfcb4f, TS 001b6ec9d9bc46c79e57a4915c44c925 -> TS 464c771a333746b6aab3fc7a84481415 move failed, destination replica disappeared from tablet's Raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:11.304872 4544 ts_tablet_manager.cc:927] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: Initiating tablet copy from peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:53:11.306025 4544 tablet_copy_client.cc:323] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:34305
I20250626 01:53:11.306828 3779 tablet_copy_service.cc:140] P 7e5284db6ec74f4a992cbd71b1dfc53f: Received BeginTabletCopySession request for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 from peer 464c771a333746b6aab3fc7a84481415 ({username='slave'} at 127.0.0.1:41826)
I20250626 01:53:11.306964 3779 tablet_copy_service.cc:161] P 7e5284db6ec74f4a992cbd71b1dfc53f: Beginning new tablet copy session on tablet 20ad5196a1fa4032afaa62ccab7e3fd1 from peer 464c771a333746b6aab3fc7a84481415 at {username='slave'} at 127.0.0.1:41826: session id = 464c771a333746b6aab3fc7a84481415-20ad5196a1fa4032afaa62ccab7e3fd1
I20250626 01:53:11.309891 3779 tablet_copy_source_session.cc:215] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:11.311160 4544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ad5196a1fa4032afaa62ccab7e3fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:11.317198 4544 tablet_copy_client.cc:806] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:11.317461 4544 tablet_copy_client.cc:670] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:11.319130 4545 tablet_copy_client.cc:1025] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:11.928268 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:11.931048 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:11.931264 3916 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:53:11.932363 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:11.932555 3916 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:11.933538 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:11.933712 3840 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:11.934087 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 01:53:11.934343 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:12.276609 4548 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:12.276782 4548 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:12.277441 3767 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:53:12.277593 3767 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:12.277695 3767 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:12.279697 3767 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:12.280278 3767 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 3 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:12.280702 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:12.280771 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:12.280887 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:12.280946 3916 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:12.281023 3916 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:12.281260 3916 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:12.281479 3916 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:12.283584 3840 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 3.
I20250626 01:53:12.284011 3735 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:12.284060 3916 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 3.
I20250626 01:53:12.284338 4547 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:12.284478 4547 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:12.284739 4547 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:12.287359 3660 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 2 to 3, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 3 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:12.357999 4514 raft_consensus.cc:991] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:12.358158 4514 raft_consensus.cc:1079] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:12.358834 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33212
I20250626 01:53:12.359015 3840 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:12.359124 3840 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:12.362319 3840 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:12.362977 3840 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 4 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:12.363360 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:12.363469 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:12.363637 3767 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:12.363656 3916 raft_consensus.cc:3053] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 01:53:12.363973 3916 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:12.364239 3916 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:12.364482 3916 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:12.366433 3767 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 4.
I20250626 01:53:12.366832 3810 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:12.367033 3916 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 4.
I20250626 01:53:12.367110 4530 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:53:12.367595 4530 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:12.368235 4530 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:12.371148 3660 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 3 to 4, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 4 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:12.525875 4554 raft_consensus.cc:991] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:12.526052 4554 raft_consensus.cc:1079] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:12.526723 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:12.526892 3916 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:12.527019 3916 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:12.529253 3916 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:12.529925 3916 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 3 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:12.530381 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:12.530558 3840 raft_consensus.cc:3053] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:12.530637 3840 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:12.530886 3840 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:12.531203 3840 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:12.533110 3840 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 3.
I20250626 01:53:12.533419 4163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
I20250626 01:53:12.533558 4163 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:12.533532 3884 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:53:12.533878 4514 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:12.534013 4514 raft_consensus.cc:695] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:12.534255 4514 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:12.536384 4163 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 3.
I20250626 01:53:12.537305 3660 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 2 to 3, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 3 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:12.795552 3767 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250626 01:53:12.796217 4554 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:12.812215 3916 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250626 01:53:12.813701 4554 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:12.837496 3916 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:12.838045 4556 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:12.842298 3840 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:12.842906 4556 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:12.915709 3840 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 01:53:12.916272 4514 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:12.920332 4163 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20250626 01:53:12.921241 4548 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:13.319532 4544 tablet_copy_client.cc:538] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:13.322849 4544 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: Bootstrap starting.
I20250626 01:53:13.364681 4544 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:13.365233 4544 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: Bootstrap complete.
I20250626 01:53:13.365581 4544 ts_tablet_manager.cc:1397] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: Time spent bootstrapping tablet: real 0.043s user 0.040s sys 0.004s
I20250626 01:53:13.366523 4544 raft_consensus.cc:357] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:13.366729 4544 raft_consensus.cc:738] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Initialized, Role: LEARNER
I20250626 01:53:13.366957 4544 consensus_queue.cc:260] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:13.367663 4544 ts_tablet_manager.cc:1428] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:53:13.368422 3779 tablet_copy_service.cc:342] P 7e5284db6ec74f4a992cbd71b1dfc53f: Request end of tablet copy session 464c771a333746b6aab3fc7a84481415-20ad5196a1fa4032afaa62ccab7e3fd1 received from {username='slave'} at 127.0.0.1:41826
I20250626 01:53:13.368587 3779 tablet_copy_service.cc:434] P 7e5284db6ec74f4a992cbd71b1dfc53f: ending tablet copy session 464c771a333746b6aab3fc7a84481415-20ad5196a1fa4032afaa62ccab7e3fd1 on tablet 20ad5196a1fa4032afaa62ccab7e3fd1 with peer 464c771a333746b6aab3fc7a84481415
I20250626 01:53:13.402488 4238 raft_consensus.cc:1215] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20250626 01:53:13.911928 4556 raft_consensus.cc:1062] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: attempting to promote NON_VOTER 464c771a333746b6aab3fc7a84481415 to VOTER
I20250626 01:53:13.912840 4556 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:13.915771 3916 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:13.915922 4238 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 LEARNER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:13.916347 3840 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20250626 01:53:13.916745 4577 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.001s
I20250626 01:53:13.917531 4556 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:13.917836 4576 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:13.934780 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:13.935498 4576 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.939275 3767 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:13.940104 3916 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.940099 4238 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.940835 3667 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:13.944368 3840 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.946609 3767 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:13.949275 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "978023fb60024f26a213be77e43d8140"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:13.949554 3918 raft_consensus.cc:604] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:13.950850 3837 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:13.951148 3837 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:13.951799 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 01:53:13.952183 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:13.959502 3767 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:13.962054 4238 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:13.962283 3840 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:13.962749 4577 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:13.963145 4576 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:13.965474 4577 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.966102 4238 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.968061 3840 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:13.969635 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:13.971100 3659 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:13.982453 3904 tablet_service.cc:1515] Processing DeleteTablet for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 with delete_type TABLET_DATA_TOMBSTONED (TS 001b6ec9d9bc46c79e57a4915c44c925 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:47144
I20250626 01:53:13.983865 4585 tablet_replica.cc:331] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:13.984159 4585 raft_consensus.cc:2241] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:13.984380 4585 raft_consensus.cc:2270] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:13.985302 4585 ts_tablet_manager.cc:1905] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:13.990855 4585 ts_tablet_manager.cc:1918] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20250626 01:53:13.991008 4585 log.cc:1199] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/wals/20ad5196a1fa4032afaa62ccab7e3fd1
I20250626 01:53:13.991838 3653 catalog_manager.cc:4928] TS 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): tablet 20ad5196a1fa4032afaa62ccab7e3fd1 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:14.250011 4573 raft_consensus.cc:991] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:14.250176 4573 raft_consensus.cc:1079] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:14.250846 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "978023fb60024f26a213be77e43d8140"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33212
I20250626 01:53:14.251039 3840 raft_consensus.cc:491] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:14.251135 3840 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:14.253101 3840 raft_consensus.cc:513] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.253746 3840 leader_election.cc:290] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 3 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:14.254199 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:14.254210 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "978023fb60024f26a213be77e43d8140" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:14.254369 3765 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:14.254400 3916 raft_consensus.cc:3053] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:14.254472 3916 raft_consensus.cc:738] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:14.254688 3916 consensus_queue.cc:260] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.254940 3916 raft_consensus.cc:3058] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:14.256614 3765 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 3.
I20250626 01:53:14.256846 3916 raft_consensus.cc:2466] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 3.
I20250626 01:53:14.257108 3810 leader_election.cc:304] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:14.257371 4570 raft_consensus.cc:2802] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:14.257720 4570 raft_consensus.cc:695] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:14.257979 4570 consensus_queue.cc:237] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.260646 3659 catalog_manager.cc:5582] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 2 to 3, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:14.345817 4576 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:14.345996 4576 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:14.346688 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33226
I20250626 01:53:14.346874 3840 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:14.346985 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:14.348982 3840 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.349635 3840 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 4 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:14.350164 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:14.350168 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:14.350313 3765 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 LEADER]: Stepping down as leader of term 3
I20250626 01:53:14.350358 3916 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:14.350385 3765 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:14.350642 3765 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.350936 3765 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:14.352450 3916 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 4.
I20250626 01:53:14.352751 3765 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 4.
I20250626 01:53:14.352926 3809 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:53:14.353247 4570 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:53:14.353397 4570 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:14.353644 4570 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.356500 3659 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 3 to 4, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 4 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:14.358335 4587 raft_consensus.cc:991] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:14.358544 4587 raft_consensus.cc:1079] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:14.359210 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:14.359438 3916 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:14.359563 3916 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:14.362100 3916 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.362854 3916 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 5 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:14.363379 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:14.363448 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:14.363588 3840 raft_consensus.cc:3053] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Stepping down as leader of term 4
I20250626 01:53:14.363597 3765 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:14.363684 3840 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:14.364002 3840 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.364295 3840 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:14.365633 3765 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 5.
I20250626 01:53:14.366045 3884 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:14.366329 4573 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 01:53:14.366379 3840 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 5.
I20250626 01:53:14.366506 4573 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:14.366798 4573 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:14.369952 3659 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 4 to 5, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:14.664988 3916 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250626 01:53:14.665592 4587 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:14.669301 3765 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 3 index: 27. (index mismatch)
I20250626 01:53:14.669868 4587 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20250626 01:53:14.862571 3765 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20250626 01:53:14.863173 4573 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:14.866621 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20250626 01:53:14.867281 4601 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:14.906211 3916 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250626 01:53:14.906828 4587 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:14.910820 3765 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20250626 01:53:14.911374 4587 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:14.976589 4238 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:14.979398 3840 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:14.979426 3918 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:14.979975 4606 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:14.980311 4597 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:14.985810 4606 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:14.986835 3918 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
W20250626 01:53:14.987434 4201 consensus_peers.cc:487] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e80ca9ec9a27407fa4e5ae542fdad35d. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:14.988621 3840 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:14.992136 3659 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 reported cstate change: config changed from index 28 to 30, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "464c771a333746b6aab3fc7a84481415" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:14.992345 3837 consensus_queue.cc:237] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 3.27, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:14.994292 3765 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
W20250626 01:53:14.994628 3810 consensus_peers.cc:487] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 978023fb60024f26a213be77e43d8140. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:14.994974 4587 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:14.995044 3918 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20250626 01:53:14.995616 4587 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20250626 01:53:14.997874 4570 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:14.998512 3765 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:14.999163 3918 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Committing config change with OpId 3.28: config changed from index -1 to 28, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:15.004320 3659 catalog_manager.cc:5582] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index -1 to 28, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:15.005529 3840 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:15.008517 3765 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 01:53:15.008517 3918 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 01:53:15.009580 4570 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:15.011046 4570 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:15.013211 4593 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
W20250626 01:53:15.017889 3810 consensus_peers.cc:487] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34f950fcb4ee4d5aad0d3f06ed2081af. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:15.017680 3918 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:15.021179 3767 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index -1 to 29, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:15.021503 3765 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:15.022048 3664 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index -1 to 29, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:53:15.029453 3732 consensus_peers.cc:487] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20ad5196a1fa4032afaa62ccab7e3fd1. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:15.029963 4238 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:15.030117 3840 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:15.030493 4576 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:15.031090 4622 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:15.034266 4625 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:15.034829 3840 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:15.034825 4238 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:15.039093 3660 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 30 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:15.040925 4163 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:15.045764 3765 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:15.046162 3918 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:15.046582 4627 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:15.047271 4627 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:15.051280 4598 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
W20250626 01:53:15.052155 4126 consensus_peers.cc:487] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ce2bf1f3ecc44f7824389e2febfcb4f. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:15.052166 3765 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:15.057627 3660 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 29 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:15.059476 3918 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:15.061957 3765 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250626 01:53:15.062009 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250626 01:53:15.062884 4586 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:15.063191 4601 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20250626 01:53:15.063589 3884 consensus_peers.cc:487] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc9d4e02a1e14b369f999b13dad44544. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:15.066368 4601 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:15.067369 3765 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:15.068176 3840 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:15.072711 3659 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index -1 to 31, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:15.075095 3918 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:15.458496 4634 ts_tablet_manager.cc:927] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: Initiating tablet copy from peer 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:15.459415 4634 tablet_copy_client.cc:323] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:40297
I20250626 01:53:15.460116 3854 tablet_copy_service.cc:140] P 311ee4b8be9f47a8a3de56840e44425a: Received BeginTabletCopySession request for tablet 34f950fcb4ee4d5aad0d3f06ed2081af from peer 464c771a333746b6aab3fc7a84481415 ({username='slave'} at 127.0.0.1:57254)
I20250626 01:53:15.460290 3854 tablet_copy_service.cc:161] P 311ee4b8be9f47a8a3de56840e44425a: Beginning new tablet copy session on tablet 34f950fcb4ee4d5aad0d3f06ed2081af from peer 464c771a333746b6aab3fc7a84481415 at {username='slave'} at 127.0.0.1:57254: session id = 464c771a333746b6aab3fc7a84481415-34f950fcb4ee4d5aad0d3f06ed2081af
I20250626 01:53:15.463090 3854 tablet_copy_source_session.cc:215] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:15.464412 4634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34f950fcb4ee4d5aad0d3f06ed2081af. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:15.470018 4634 tablet_copy_client.cc:806] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:15.470346 4634 tablet_copy_client.cc:670] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:15.472257 4635 tablet_copy_client.cc:1025] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:15.489387 4636 ts_tablet_manager.cc:927] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: Initiating tablet copy from peer 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:15.490116 4636 tablet_copy_client.cc:323] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:40297
I20250626 01:53:15.490731 3854 tablet_copy_service.cc:140] P 311ee4b8be9f47a8a3de56840e44425a: Received BeginTabletCopySession request for tablet 978023fb60024f26a213be77e43d8140 from peer fb97ea56eeb04a3d9bf983411d630ee7 ({username='slave'} at 127.0.0.1:33268)
I20250626 01:53:15.490928 3854 tablet_copy_service.cc:161] P 311ee4b8be9f47a8a3de56840e44425a: Beginning new tablet copy session on tablet 978023fb60024f26a213be77e43d8140 from peer fb97ea56eeb04a3d9bf983411d630ee7 at {username='slave'} at 127.0.0.1:33268: session id = fb97ea56eeb04a3d9bf983411d630ee7-978023fb60024f26a213be77e43d8140
I20250626 01:53:15.493222 3854 tablet_copy_source_session.cc:215] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:15.494362 4636 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978023fb60024f26a213be77e43d8140. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:15.500602 4636 tablet_copy_client.cc:806] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:15.500972 4636 tablet_copy_client.cc:670] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:15.502429 4637 tablet_copy_client.cc:1025] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:15.537670 4638 ts_tablet_manager.cc:927] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: Initiating tablet copy from peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:53:15.538516 4638 tablet_copy_client.cc:323] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:34305
I20250626 01:53:15.539227 3779 tablet_copy_service.cc:140] P 7e5284db6ec74f4a992cbd71b1dfc53f: Received BeginTabletCopySession request for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 from peer 9832bf960c714aff8c1dc4fc62ee4099 ({username='slave'} at 127.0.0.1:41818)
I20250626 01:53:15.539377 3779 tablet_copy_service.cc:161] P 7e5284db6ec74f4a992cbd71b1dfc53f: Beginning new tablet copy session on tablet 20ad5196a1fa4032afaa62ccab7e3fd1 from peer 9832bf960c714aff8c1dc4fc62ee4099 at {username='slave'} at 127.0.0.1:41818: session id = 9832bf960c714aff8c1dc4fc62ee4099-20ad5196a1fa4032afaa62ccab7e3fd1
I20250626 01:53:15.539424 4640 ts_tablet_manager.cc:927] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: Initiating tablet copy from peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:15.540283 4640 tablet_copy_client.cc:323] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35233
I20250626 01:53:15.542430 3779 tablet_copy_source_session.cc:215] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:15.543532 3928 tablet_copy_service.cc:140] P 001b6ec9d9bc46c79e57a4915c44c925: Received BeginTabletCopySession request for tablet bc9d4e02a1e14b369f999b13dad44544 from peer fb97ea56eeb04a3d9bf983411d630ee7 ({username='slave'} at 127.0.0.1:47200)
I20250626 01:53:15.543702 3928 tablet_copy_service.cc:161] P 001b6ec9d9bc46c79e57a4915c44c925: Beginning new tablet copy session on tablet bc9d4e02a1e14b369f999b13dad44544 from peer fb97ea56eeb04a3d9bf983411d630ee7 at {username='slave'} at 127.0.0.1:47200: session id = fb97ea56eeb04a3d9bf983411d630ee7-bc9d4e02a1e14b369f999b13dad44544
I20250626 01:53:15.544261 4638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20ad5196a1fa4032afaa62ccab7e3fd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:15.546680 3928 tablet_copy_source_session.cc:215] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:15.547950 4640 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d4e02a1e14b369f999b13dad44544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:15.550607 4638 tablet_copy_client.cc:806] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:15.550927 4638 tablet_copy_client.cc:670] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:15.554517 4640 tablet_copy_client.cc:806] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:15.554785 4640 tablet_copy_client.cc:670] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:15.556443 4639 tablet_copy_client.cc:1025] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:15.556613 4641 tablet_copy_client.cc:1025] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:15.584578 4642 ts_tablet_manager.cc:927] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: Initiating tablet copy from peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:15.585230 4642 tablet_copy_client.cc:323] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:45827
I20250626 01:53:15.592026 4248 tablet_copy_service.cc:140] P 464c771a333746b6aab3fc7a84481415: Received BeginTabletCopySession request for tablet e80ca9ec9a27407fa4e5ae542fdad35d from peer 9832bf960c714aff8c1dc4fc62ee4099 ({username='slave'} at 127.0.0.1:60218)
I20250626 01:53:15.592288 4248 tablet_copy_service.cc:161] P 464c771a333746b6aab3fc7a84481415: Beginning new tablet copy session on tablet e80ca9ec9a27407fa4e5ae542fdad35d from peer 9832bf960c714aff8c1dc4fc62ee4099 at {username='slave'} at 127.0.0.1:60218: session id = 9832bf960c714aff8c1dc4fc62ee4099-e80ca9ec9a27407fa4e5ae542fdad35d
I20250626 01:53:15.595239 4248 tablet_copy_source_session.cc:215] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:15.596694 4642 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e80ca9ec9a27407fa4e5ae542fdad35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:15.603235 4642 tablet_copy_client.cc:806] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:15.603614 4642 tablet_copy_client.cc:670] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:15.605538 4643 tablet_copy_client.cc:1025] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:15.662268 4646 ts_tablet_manager.cc:927] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: Initiating tablet copy from peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:15.663100 4646 tablet_copy_client.cc:323] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:44397
I20250626 01:53:15.670008 4173 tablet_copy_service.cc:140] P fb97ea56eeb04a3d9bf983411d630ee7: Received BeginTabletCopySession request for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f from peer 9832bf960c714aff8c1dc4fc62ee4099 ({username='slave'} at 127.0.0.1:48682)
I20250626 01:53:15.670246 4173 tablet_copy_service.cc:161] P fb97ea56eeb04a3d9bf983411d630ee7: Beginning new tablet copy session on tablet 0ce2bf1f3ecc44f7824389e2febfcb4f from peer 9832bf960c714aff8c1dc4fc62ee4099 at {username='slave'} at 127.0.0.1:48682: session id = 9832bf960c714aff8c1dc4fc62ee4099-0ce2bf1f3ecc44f7824389e2febfcb4f
I20250626 01:53:15.672705 4173 tablet_copy_source_session.cc:215] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:15.673868 4646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:15.680063 4646 tablet_copy_client.cc:806] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:15.680387 4646 tablet_copy_client.cc:670] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:15.682168 4647 tablet_copy_client.cc:1025] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:15.952625 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:15.956297 3837 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:15.956543 3837 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:15.957660 3837 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:15.957841 3837 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:53:15.959187 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:15.959410 3918 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Received request to transfer leadership to TS fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:15.959875 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20250626 01:53:15.960155 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:15.961951 4587 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:15.962105 4587 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:15.963100 3918 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:15.963439 3918 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:15.963593 3918 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:15.966295 3918 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:15.967031 3918 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 5 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:15.967689 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:15.967689 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:15.967988 3767 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:15.968044 3837 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Stepping down as leader of term 4
I20250626 01:53:15.968129 3837 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:15.968441 3837 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:15.968775 3837 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:15.970937 3767 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 5.
I20250626 01:53:15.971541 3884 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:15.971360 3837 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 5.
I20250626 01:53:15.971976 4586 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Leader election won for term 5
I20250626 01:53:15.972558 4586 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:15.972905 4586 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:15.976104 3664 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 4 to 5, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:16.078155 4649 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:16.078358 4649 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:16.078954 3767 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41788
I20250626 01:53:16.079160 3767 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:16.079286 3767 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:16.081300 3767 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:16.082206 3767 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 4 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:16.082474 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:16.082690 3918 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:16.084095 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:16.084282 3837 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Stepping down as leader of term 3
I20250626 01:53:16.084389 3837 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:16.084718 3918 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 4.
I20250626 01:53:16.084677 3837 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:16.085045 3837 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:16.085165 3734 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:16.085484 4652 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:53:16.085624 4652 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:16.085887 4652 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:16.087519 3837 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 4.
I20250626 01:53:16.089033 3664 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 3 to 4, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 4 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:16.090719 4660 raft_consensus.cc:991] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:16.090895 4660 raft_consensus.cc:1079] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Signalling peer fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:16.091591 4162 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:54818
I20250626 01:53:16.091854 4162 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:16.092072 4162 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:16.094587 4162 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:16.095427 4162 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 4 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:16.095871 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:16.096105 3918 raft_consensus.cc:3053] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Stepping down as leader of term 3
I20250626 01:53:16.096060 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:16.096204 3918 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:16.096271 3837 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:16.096501 3918 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:16.096839 3918 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Advancing to term 4
I20250626 01:53:16.098991 3837 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 4.
I20250626 01:53:16.099136 3918 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 4.
I20250626 01:53:16.099102 3767 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:16.099381 3767 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Received request to transfer leadership
I20250626 01:53:16.099511 4129 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, fb97ea56eeb04a3d9bf983411d630ee7; no voters:
I20250626 01:53:16.099861 4662 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 FOLLOWER]: Leader election won for term 4
I20250626 01:53:16.100064 4662 raft_consensus.cc:695] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 LEADER]: Becoming Leader. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Running, Role: LEADER
I20250626 01:53:16.100416 4662 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:16.105000 3664 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: term changed from 3 to 4, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68). New cstate: current_term: 4 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:16.108577 3767 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:16.108816 3767 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Received request to transfer leadership
W20250626 01:53:16.109218 3654 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 34f950fcb4ee4d5aad0d3f06ed2081af already in progress [suppressed 2 similar messages]
I20250626 01:53:16.110271 3767 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:16.110440 3767 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Received request to transfer leadership
W20250626 01:53:16.110913 3715 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 34f950fcb4ee4d5aad0d3f06ed2081af already in progress
W20250626 01:53:16.110993 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 34f950fcb4ee4d5aad0d3f06ed2081af already in progress
I20250626 01:53:16.485085 3918 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 01:53:16.485587 4662 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:16.489612 3837 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20250626 01:53:16.491284 4662 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:16.567574 3837 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20250626 01:53:16.568257 4671 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:16.572681 3767 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20250626 01:53:16.573793 4653 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:16.650964 3837 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250626 01:53:16.651494 4651 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:16.653776 4652 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:16.654088 4625 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:16.656423 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33226
I20250626 01:53:16.656883 3840 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:16.657006 3840 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Advancing to term 5
W20250626 01:53:16.657982 3735 consensus_peers.cc:487] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:16.659828 3840 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:16.661490 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:16.661711 3767 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Stepping down as leader of term 4
I20250626 01:53:16.661806 3767 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:16.662104 3767 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:16.662434 3767 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:16.664222 3840 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 5 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:16.664566 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 5 candidate_status { last_received { term: 4 index: 30 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:16.655989 3918 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250626 01:53:16.665319 3767 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 5.
I20250626 01:53:16.665648 3810 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters: 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:16.665937 4649 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Leader election won for term 5
I20250626 01:53:16.666100 4649 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:16.666379 4649 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 4.30, Last appended by leader: 30, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:16.668938 3664 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 4 to 5, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 5 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:53:17.049517 3810 consensus_peers.cc:487] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:17.114254 3765 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 4 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250626 01:53:17.114815 4649 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:17.119851 3918 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:17.122419 3918 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20250626 01:53:17.122999 4587 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:17.472692 4634 tablet_copy_client.cc:538] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:17.475713 4634 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: Bootstrap starting.
I20250626 01:53:17.502846 4636 tablet_copy_client.cc:538] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:17.505868 4636 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap starting.
I20250626 01:53:17.510650 4634 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: Bootstrap replayed 1/1 log segments. Stats: ops{read=29 overwritten=0 applied=29 ignored=0} inserts{seen=116 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:17.511142 4634 tablet_bootstrap.cc:492] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: Bootstrap complete.
I20250626 01:53:17.511417 4634 ts_tablet_manager.cc:1397] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20250626 01:53:17.512262 4634 raft_consensus.cc:357] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:17.512430 4634 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Initialized, Role: LEARNER
I20250626 01:53:17.512622 4634 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 29, Last appended: 3.29, Last appended by leader: 29, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:17.513181 4634 ts_tablet_manager.cc:1428] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20250626 01:53:17.513803 3854 tablet_copy_service.cc:342] P 311ee4b8be9f47a8a3de56840e44425a: Request end of tablet copy session 464c771a333746b6aab3fc7a84481415-34f950fcb4ee4d5aad0d3f06ed2081af received from {username='slave'} at 127.0.0.1:57254
I20250626 01:53:17.513936 3854 tablet_copy_service.cc:434] P 311ee4b8be9f47a8a3de56840e44425a: ending tablet copy session 464c771a333746b6aab3fc7a84481415-34f950fcb4ee4d5aad0d3f06ed2081af on tablet 34f950fcb4ee4d5aad0d3f06ed2081af with peer 464c771a333746b6aab3fc7a84481415
I20250626 01:53:17.547860 4636 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=117 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:17.548393 4636 tablet_bootstrap.cc:492] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap complete.
I20250626 01:53:17.548661 4636 ts_tablet_manager.cc:1397] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: Time spent bootstrapping tablet: real 0.043s user 0.037s sys 0.004s
I20250626 01:53:17.549463 4636 raft_consensus.cc:357] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:17.549669 4636 raft_consensus.cc:738] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Initialized, Role: LEARNER
I20250626 01:53:17.549849 4636 consensus_queue.cc:260] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:17.550307 4636 ts_tablet_manager.cc:1428] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:53:17.550956 3854 tablet_copy_service.cc:342] P 311ee4b8be9f47a8a3de56840e44425a: Request end of tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-978023fb60024f26a213be77e43d8140 received from {username='slave'} at 127.0.0.1:33268
I20250626 01:53:17.551092 3854 tablet_copy_service.cc:434] P 311ee4b8be9f47a8a3de56840e44425a: ending tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-978023fb60024f26a213be77e43d8140 on tablet 978023fb60024f26a213be77e43d8140 with peer fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:17.556931 4640 tablet_copy_client.cc:538] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:17.557056 4638 tablet_copy_client.cc:538] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:17.560722 4638 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap starting.
I20250626 01:53:17.560722 4640 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap starting.
I20250626 01:53:17.570662 4237 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 3 LEARNER]: Advancing to term 5
I20250626 01:53:17.574813 4237 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 5 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 4 index: 30. (index mismatch)
I20250626 01:53:17.575551 4682 consensus_queue.cc:1037] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:17.603374 4682 raft_consensus.cc:1062] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: attempting to promote NON_VOTER 464c771a333746b6aab3fc7a84481415 to VOTER
I20250626 01:53:17.604190 4682 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 30, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:17.606307 4642 tablet_copy_client.cc:538] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:17.606943 3916 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250626 01:53:17.607430 4649 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:17.609877 4642 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap starting.
I20250626 01:53:17.610489 4638 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:17.610954 4638 tablet_bootstrap.cc:492] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap complete.
I20250626 01:53:17.611198 4638 ts_tablet_manager.cc:1397] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: Time spent bootstrapping tablet: real 0.051s user 0.037s sys 0.008s
I20250626 01:53:17.611979 4638 raft_consensus.cc:357] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:17.612198 4638 raft_consensus.cc:738] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Initialized, Role: LEARNER
I20250626 01:53:17.612380 4638 consensus_queue.cc:260] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:17.612823 3765 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250626 01:53:17.612941 4638 ts_tablet_manager.cc:1428] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20250626 01:53:17.613528 3779 tablet_copy_service.cc:342] P 7e5284db6ec74f4a992cbd71b1dfc53f: Request end of tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-20ad5196a1fa4032afaa62ccab7e3fd1 received from {username='slave'} at 127.0.0.1:41818
I20250626 01:53:17.613641 3779 tablet_copy_service.cc:434] P 7e5284db6ec74f4a992cbd71b1dfc53f: ending tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-20ad5196a1fa4032afaa62ccab7e3fd1 on tablet 20ad5196a1fa4032afaa62ccab7e3fd1 with peer 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:17.614003 4691 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:17.616777 4649 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:17.618631 4640 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:17.619009 4640 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap complete.
I20250626 01:53:17.619266 4640 ts_tablet_manager.cc:1397] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: Time spent bootstrapping tablet: real 0.059s user 0.036s sys 0.006s
I20250626 01:53:17.619947 4640 raft_consensus.cc:357] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:17.620260 4640 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Initialized, Role: LEARNER
I20250626 01:53:17.620496 4640 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:17.621094 4640 ts_tablet_manager.cc:1428] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250626 01:53:17.621134 4237 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 5 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250626 01:53:17.621794 3928 tablet_copy_service.cc:342] P 001b6ec9d9bc46c79e57a4915c44c925: Request end of tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-bc9d4e02a1e14b369f999b13dad44544 received from {username='slave'} at 127.0.0.1:47200
I20250626 01:53:17.621910 3928 tablet_copy_service.cc:434] P 001b6ec9d9bc46c79e57a4915c44c925: ending tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-bc9d4e02a1e14b369f999b13dad44544 on tablet bc9d4e02a1e14b369f999b13dad44544 with peer fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:17.623467 3918 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:17.624354 4682 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:17.625751 4238 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:17.627810 3765 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 29 to 32, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:17.631592 3664 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 29 to 32, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:17.667016 4642 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:17.667447 4642 tablet_bootstrap.cc:492] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap complete.
I20250626 01:53:17.667723 4642 ts_tablet_manager.cc:1397] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: Time spent bootstrapping tablet: real 0.058s user 0.032s sys 0.006s
I20250626 01:53:17.668565 4642 raft_consensus.cc:357] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:17.668738 4642 raft_consensus.cc:738] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Initialized, Role: LEARNER
I20250626 01:53:17.668952 4642 consensus_queue.cc:260] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:17.669417 4642 ts_tablet_manager.cc:1428] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20250626 01:53:17.670058 4248 tablet_copy_service.cc:342] P 464c771a333746b6aab3fc7a84481415: Request end of tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-e80ca9ec9a27407fa4e5ae542fdad35d received from {username='slave'} at 127.0.0.1:60218
I20250626 01:53:17.670200 4248 tablet_copy_service.cc:434] P 464c771a333746b6aab3fc7a84481415: ending tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-e80ca9ec9a27407fa4e5ae542fdad35d on tablet e80ca9ec9a27407fa4e5ae542fdad35d with peer 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:17.682683 4646 tablet_copy_client.cc:538] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:17.685647 4646 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap starting.
I20250626 01:53:17.688872 4311 raft_consensus.cc:1215] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.30->[2.31-2.31] Dedup: 2.31->[]
I20250626 01:53:17.725137 4646 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=106 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:17.725683 4646 tablet_bootstrap.cc:492] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap complete.
I20250626 01:53:17.726037 4646 ts_tablet_manager.cc:1397] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: Time spent bootstrapping tablet: real 0.040s user 0.041s sys 0.000s
I20250626 01:53:17.726774 4646 raft_consensus.cc:357] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:17.726966 4646 raft_consensus.cc:738] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Initialized, Role: LEARNER
I20250626 01:53:17.727186 4646 consensus_queue.cc:260] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:17.727681 4646 ts_tablet_manager.cc:1428] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:53:17.728435 4173 tablet_copy_service.cc:342] P fb97ea56eeb04a3d9bf983411d630ee7: Request end of tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-0ce2bf1f3ecc44f7824389e2febfcb4f received from {username='slave'} at 127.0.0.1:48682
I20250626 01:53:17.728581 4173 tablet_copy_service.cc:434] P fb97ea56eeb04a3d9bf983411d630ee7: ending tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-0ce2bf1f3ecc44f7824389e2febfcb4f on tablet 0ce2bf1f3ecc44f7824389e2febfcb4f with peer 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:17.943358 4162 raft_consensus.cc:1215] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.30->[5.31-5.31] Dedup: 5.31->[]
I20250626 01:53:17.960472 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:17.963811 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:17.964087 3918 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:53:17.965077 3837 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:17.965276 3837 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:17.965735 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 01:53:17.965945 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:17.973912 4162 raft_consensus.cc:1215] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.27->[3.28-3.28] Dedup: 3.28->[]
W20250626 01:53:18.061348 3654 catalog_manager.cc:4671] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): ChangeConfig:REMOVE_PEER RPC failed for tablet 34f950fcb4ee4d5aad0d3f06ed2081af: Remote error: Service unavailable: leader transfer in progress
I20250626 01:53:18.135917 4311 raft_consensus.cc:1215] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.30->[2.31-2.31] Dedup: 2.31->[]
I20250626 01:53:18.177305 4700 raft_consensus.cc:1062] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: attempting to promote NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 to VOTER
I20250626 01:53:18.178435 4700 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:18.181042 4311 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.181326 4237 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.182260 3837 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.181725 4699 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.001s
I20250626 01:53:18.182780 4651 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.183007 4702 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.200646 4692 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:18.200781 4692 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:18.200271 4651 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.200901 3837 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.201490 3918 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:18.201648 3918 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:18.201749 3918 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.203918 3918 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:18.206691 3767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:18.206881 3767 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.207639 3664 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:18.210175 4311 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.210415 4237 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.211318 4312 raft_consensus.cc:1215] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20250626 01:53:18.214442 3767 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 6.
I20250626 01:53:18.215564 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:18.215827 3918 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 6 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:18.215799 3837 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Stepping down as leader of term 5
I20250626 01:53:18.217001 3837 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:18.217320 3837 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:18.216078 4237 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 6 candidate_status { last_received { term: 5 index: 32 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:18.217650 4237 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.217715 3837 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.219908 3837 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 6.
I20250626 01:53:18.220413 3884 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:18.220652 4660 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 01:53:18.221092 4660 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:18.221357 4660 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 32, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:18.221966 4237 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 6.
I20250626 01:53:18.225485 3665 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 5 to 6, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 6 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:18.229214 3767 consensus_queue.cc:237] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:18.230966 4311 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 01:53:18.231094 4237 raft_consensus.cc:1273] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 01:53:18.231587 4702 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.231848 4699 consensus_queue.cc:1035] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.234505 4702 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.235600 4311 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.239076 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:18.239578 4238 raft_consensus.cc:2953] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.240058 3665 catalog_manager.cc:5582] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: config changed from index 32 to 33, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:18.244758 3826 tablet_service.cc:1515] Processing DeleteTablet for tablet 20ad5196a1fa4032afaa62ccab7e3fd1 with delete_type TABLET_DATA_TOMBSTONED (TS 311ee4b8be9f47a8a3de56840e44425a not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:18.245352 4710 tablet_replica.cc:331] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:18.245538 4710 raft_consensus.cc:2241] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:18.245807 4710 raft_consensus.cc:2270] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:18.246558 4710 ts_tablet_manager.cc:1905] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:18.252369 4710 ts_tablet_manager.cc:1918] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20250626 01:53:18.252518 4710 log.cc:1199] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 311ee4b8be9f47a8a3de56840e44425a: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/wals/20ad5196a1fa4032afaa62ccab7e3fd1
I20250626 01:53:18.253144 3654 catalog_manager.cc:4928] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): tablet 20ad5196a1fa4032afaa62ccab7e3fd1 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
W20250626 01:53:18.282546 3653 catalog_manager.cc:5112] ChangeConfig:REMOVE_PEER RPC for tablet 34f950fcb4ee4d5aad0d3f06ed2081af with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER failed with leader 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20250626 01:53:18.317584 4711 raft_consensus.cc:1062] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: attempting to promote NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 to VOTER
I20250626 01:53:18.318755 4711 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:18.321218 3837 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250626 01:53:18.321283 3767 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250626 01:53:18.321318 4162 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 LEARNER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20250626 01:53:18.321852 4660 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.322247 4711 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.322628 4713 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.326506 4711 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 31 to 32, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.330353 3837 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.331544 4163 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.332198 3767 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.335479 3667 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 31 to 32, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:18.343381 3918 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:18.343658 4713 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:18.343789 4713 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:18.344383 3767 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:53:18.344547 3767 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:18.344637 3767 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.345356 4163 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250626 01:53:18.345582 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20250626 01:53:18.346199 4660 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.346697 4660 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.347692 3767 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:18.348433 3767 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 6 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:18.348601 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:18.349699 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.350026 3837 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.352566 3840 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 6.
I20250626 01:53:18.352990 3735 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:18.348726 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 6 candidate_status { last_received { term: 5 index: 30 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:18.353279 4651 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Leader election won for term 6
I20250626 01:53:18.353320 3918 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 01:53:18.353377 3918 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:18.353399 4651 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:18.353621 4651 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:18.353586 3918 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 30, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:18.353889 3918 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:18.348984 4660 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.357667 3918 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 6.
I20250626 01:53:18.358844 3665 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 32 to 33, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.360842 3653 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bc9d4e02a1e14b369f999b13dad44544 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:18.362224 4163 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.363866 3754 tablet_service.cc:1515] Processing DeleteTablet for tablet bc9d4e02a1e14b369f999b13dad44544 with delete_type TABLET_DATA_TOMBSTONED (TS 7e5284db6ec74f4a992cbd71b1dfc53f not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:18.369719 3660 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 5 to 6, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 6 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:18.380350 4718 tablet_replica.cc:331] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:18.380563 4718 raft_consensus.cc:2241] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:18.380867 4718 raft_consensus.cc:2270] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:18.382035 4718 ts_tablet_manager.cc:1905] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:18.387773 4718 ts_tablet_manager.cc:1918] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20250626 01:53:18.387976 4718 log.cc:1199] T bc9d4e02a1e14b369f999b13dad44544 P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/wals/bc9d4e02a1e14b369f999b13dad44544
I20250626 01:53:18.388710 3654 catalog_manager.cc:4928] TS 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): tablet bc9d4e02a1e14b369f999b13dad44544 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:18.553519 4720 raft_consensus.cc:1062] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: attempting to promote NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 to VOTER
I20250626 01:53:18.554409 4720 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:18.557044 3916 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.557044 4311 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.557261 3767 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.557679 4720 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.558043 4719 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.558460 4631 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.565954 4719 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.567263 3767 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.569733 4311 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.570842 3660 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:18.572713 4692 raft_consensus.cc:1062] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: attempting to promote NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 to VOTER
I20250626 01:53:18.573036 3916 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.573597 4692 consensus_queue.cc:237] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 26, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:18.578361 4163 consensus_queue.cc:237] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:18.581393 4311 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 01:53:18.583348 4631 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.583657 4163 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 01:53:18.583828 3765 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 01:53:18.584398 4587 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:18.585281 3767 raft_consensus.cc:1273] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 01:53:18.585638 4682 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:18.585798 4631 consensus_queue.cc:1035] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.584074 3916 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20250626 01:53:18.587755 4682 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20250626 01:53:18.588474 4719 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.589021 3766 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.592094 3714 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:18.595687 3665 catalog_manager.cc:5582] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 32 to 33, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:18.601486 4312 raft_consensus.cc:2953] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.611442 4682 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Committing config change with OpId 3.29: config changed from index 28 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.612533 3916 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.616086 4163 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.618682 3765 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Committing config change with OpId 3.29: config changed from index 28 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.620342 3660 catalog_manager.cc:5582] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 28 to 29, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.621277 3904 tablet_service.cc:1515] Processing DeleteTablet for tablet 0ce2bf1f3ecc44f7824389e2febfcb4f with delete_type TABLET_DATA_TOMBSTONED (TS 001b6ec9d9bc46c79e57a4915c44c925 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:47144
I20250626 01:53:18.627542 3840 consensus_queue.cc:237] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 26, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:18.633363 3765 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 01:53:18.633363 4163 raft_consensus.cc:1273] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20250626 01:53:18.635536 4587 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:18.636125 4682 consensus_queue.cc:1035] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:18.637564 4730 tablet_replica.cc:331] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:18.637892 4730 raft_consensus.cc:2241] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:18.638231 4730 raft_consensus.cc:2270] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:18.639336 4730 ts_tablet_manager.cc:1905] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:18.642093 4587 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.643240 4163 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.646497 3765 raft_consensus.cc:2953] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:18.647660 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 978023fb60024f26a213be77e43d8140 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:18.646981 4730 ts_tablet_manager.cc:1918] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20250626 01:53:18.648094 4730 log.cc:1199] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 001b6ec9d9bc46c79e57a4915c44c925: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/wals/0ce2bf1f3ecc44f7824389e2febfcb4f
I20250626 01:53:18.648950 3653 catalog_manager.cc:4928] TS 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): tablet 0ce2bf1f3ecc44f7824389e2febfcb4f (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:18.649325 3667 catalog_manager.cc:5582] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 29 to 30, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:18.656136 3904 tablet_service.cc:1515] Processing DeleteTablet for tablet 978023fb60024f26a213be77e43d8140 with delete_type TABLET_DATA_TOMBSTONED (TS 001b6ec9d9bc46c79e57a4915c44c925 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:47144
I20250626 01:53:18.658087 4730 tablet_replica.cc:331] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:18.658510 3765 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20250626 01:53:18.658946 4730 raft_consensus.cc:2241] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:18.659301 4730 raft_consensus.cc:2270] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:18.659430 4709 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.661453 4730 ts_tablet_manager.cc:1905] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:18.668721 4730 ts_tablet_manager.cc:1918] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.29
I20250626 01:53:18.668954 4730 log.cc:1199] T 978023fb60024f26a213be77e43d8140 P 001b6ec9d9bc46c79e57a4915c44c925: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/wals/978023fb60024f26a213be77e43d8140
I20250626 01:53:18.678411 3840 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20250626 01:53:18.679128 4709 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.690969 3653 catalog_manager.cc:4928] TS 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): tablet 978023fb60024f26a213be77e43d8140 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:18.692207 4237 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 6 index: 33. (index mismatch)
I20250626 01:53:18.692868 4713 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:18.716944 3916 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 6.33, Last appended by leader: 32, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:18.718851 4238 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20250626 01:53:18.718973 3840 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20250626 01:53:18.719518 4713 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:18.719777 4711 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:18.722306 4709 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Committing config change with OpId 6.34: config changed from index 32 to 34, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:18.724520 4238 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 32 to 34, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:18.726868 3653 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 34f950fcb4ee4d5aad0d3f06ed2081af with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20250626 01:53:18.726408 3840 raft_consensus.cc:2953] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Committing config change with OpId 6.34: config changed from index 32 to 34, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:18.729162 3665 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 reported cstate change: config changed from index 32 to 34, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New cstate: current_term: 6 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:18.732860 3754 tablet_service.cc:1515] Processing DeleteTablet for tablet 34f950fcb4ee4d5aad0d3f06ed2081af with delete_type TABLET_DATA_TOMBSTONED (TS 7e5284db6ec74f4a992cbd71b1dfc53f not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:18.733250 4718 tablet_replica.cc:331] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:18.733458 4718 raft_consensus.cc:2241] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:18.733816 4718 raft_consensus.cc:2270] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:18.734809 4718 ts_tablet_manager.cc:1905] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:18.741282 4718 ts_tablet_manager.cc:1918] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.33
I20250626 01:53:18.741432 4718 log.cc:1199] T 34f950fcb4ee4d5aad0d3f06ed2081af P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/wals/34f950fcb4ee4d5aad0d3f06ed2081af
I20250626 01:53:18.742158 3654 catalog_manager.cc:4928] TS 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): tablet 34f950fcb4ee4d5aad0d3f06ed2081af (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:18.800946 4676 raft_consensus.cc:1062] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: attempting to promote NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 to VOTER
I20250626 01:53:18.801855 4676 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:18.802035 3916 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20250626 01:53:18.802551 4651 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:18.804507 3918 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:18.804491 4312 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEARNER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:18.805181 4741 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:18.805537 4739 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:18.809777 3840 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:18.809557 4741 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.812479 4676 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:18.813232 4312 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.814496 3918 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.821062 3837 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 6 index: 31. (index mismatch)
I20250626 01:53:18.821894 4699 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:18.819726 3660 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: config changed from index 30 to 31, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "464c771a333746b6aab3fc7a84481415" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.821945 3840 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.830838 4238 consensus_queue.cc:237] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:18.834806 4312 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.835510 4739 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.836181 3916 raft_consensus.cc:1273] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:18.838238 4739 consensus_queue.cc:1035] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:18.838738 4745 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.839622 4312 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.842633 3714 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet e80ca9ec9a27407fa4e5ae542fdad35d with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:18.844216 3665 catalog_manager.cc:5582] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 reported cstate change: config changed from index 31 to 32, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "464c771a333746b6aab3fc7a84481415" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:18.846423 3916 raft_consensus.cc:2953] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:18.847996 3826 tablet_service.cc:1515] Processing DeleteTablet for tablet e80ca9ec9a27407fa4e5ae542fdad35d with delete_type TABLET_DATA_TOMBSTONED (TS 311ee4b8be9f47a8a3de56840e44425a not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:18.848745 4748 tablet_replica.cc:331] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:18.848945 4748 raft_consensus.cc:2241] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:18.849151 4748 raft_consensus.cc:2270] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:18.850152 4748 ts_tablet_manager.cc:1905] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:18.855238 4748 ts_tablet_manager.cc:1918] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250626 01:53:18.855363 4748 log.cc:1199] T e80ca9ec9a27407fa4e5ae542fdad35d P 311ee4b8be9f47a8a3de56840e44425a: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/wals/e80ca9ec9a27407fa4e5ae542fdad35d
I20250626 01:53:18.856005 3654 catalog_manager.cc:4928] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): tablet e80ca9ec9a27407fa4e5ae542fdad35d (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:19.849750 4163 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:19.852880 3916 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250626 01:53:19.853379 4720 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:19.854171 3837 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 4 index: 33. (index mismatch)
I20250626 01:53:19.854718 4720 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:19.857107 4753 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 LEADER]: Committing config change with OpId 4.33: config changed from index 30 to 33, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
W20250626 01:53:19.859831 4129 consensus_peers.cc:487] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eec015ad4dee46efb54559801ff9896b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:19.859596 3916 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:19.860985 3837 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Committing config change with OpId 4.33: config changed from index 30 to 33, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:19.862870 3665 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 30 to 33, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New cstate: current_term: 4 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:19.864480 3840 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:19.869120 4163 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 01:53:19.869939 4760 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20250626 01:53:19.870409 3810 consensus_peers.cc:487] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a -> Peer 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827): Couldn't send request to peer 464c771a333746b6aab3fc7a84481415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71684e4b73c24f57b67e052454de9097. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:19.871316 3765 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20250626 01:53:19.872073 4682 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:19.876089 4760 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:19.877282 4163 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:19.877928 3765 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } } }
I20250626 01:53:19.881929 3660 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 29 to 30, NON_VOTER 464c771a333746b6aab3fc7a84481415 (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:19.896234 4312 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:19.898732 3916 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20250626 01:53:19.898732 3840 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
W20250626 01:53:19.899124 4278 consensus_peers.cc:487] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 -> Peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397): Couldn't send request to peer fb97ea56eeb04a3d9bf983411d630ee7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 96fb20902f87437487c5f36d71de0b9b. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:19.899359 4770 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:19.899639 4769 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20250626 01:53:19.902330 4769 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:19.903007 3840 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:19.903326 3916 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } } }
I20250626 01:53:19.906950 3665 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: config changed from index 28 to 30, NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "9832bf960c714aff8c1dc4fc62ee4099" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:19.907435 3916 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:19.912285 4163 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250626 01:53:19.913139 4775 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:19.914356 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250626 01:53:19.914826 4775 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:19.916738 4775 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:19.920774 4163 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
W20250626 01:53:19.921815 3881 consensus_peers.cc:487] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc9d4e02a1e14b369f999b13dad44544. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:19.922098 3660 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 33 to 34, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New cstate: current_term: 5 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:19.925932 3840 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Committing config change with OpId 5.34: config changed from index 33 to 34, NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) added. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } } }
I20250626 01:53:19.966305 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:19.969300 3765 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:19.969545 3765 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:19.970664 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:19.970855 3916 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:19.971472 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 01:53:19.971742 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:20.266428 4779 ts_tablet_manager.cc:927] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: Initiating tablet copy from peer fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:20.267083 4779 tablet_copy_client.cc:323] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:44397
I20250626 01:53:20.274364 4173 tablet_copy_service.cc:140] P fb97ea56eeb04a3d9bf983411d630ee7: Received BeginTabletCopySession request for tablet eec015ad4dee46efb54559801ff9896b from peer 464c771a333746b6aab3fc7a84481415 ({username='slave'} at 127.0.0.1:48690)
I20250626 01:53:20.274537 4173 tablet_copy_service.cc:161] P fb97ea56eeb04a3d9bf983411d630ee7: Beginning new tablet copy session on tablet eec015ad4dee46efb54559801ff9896b from peer 464c771a333746b6aab3fc7a84481415 at {username='slave'} at 127.0.0.1:48690: session id = 464c771a333746b6aab3fc7a84481415-eec015ad4dee46efb54559801ff9896b
I20250626 01:53:20.276743 4173 tablet_copy_source_session.cc:215] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:20.278020 4779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec015ad4dee46efb54559801ff9896b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:20.283763 4779 tablet_copy_client.cc:806] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:20.284041 4779 tablet_copy_client.cc:670] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:20.285842 4780 tablet_copy_client.cc:1025] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:20.292827 4783 ts_tablet_manager.cc:927] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: Initiating tablet copy from peer 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:20.293473 4783 tablet_copy_client.cc:323] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35233
I20250626 01:53:20.294106 3928 tablet_copy_service.cc:140] P 001b6ec9d9bc46c79e57a4915c44c925: Received BeginTabletCopySession request for tablet bc9d4e02a1e14b369f999b13dad44544 from peer 9832bf960c714aff8c1dc4fc62ee4099 ({username='slave'} at 127.0.0.1:60820)
I20250626 01:53:20.294246 3928 tablet_copy_service.cc:161] P 001b6ec9d9bc46c79e57a4915c44c925: Beginning new tablet copy session on tablet bc9d4e02a1e14b369f999b13dad44544 from peer 9832bf960c714aff8c1dc4fc62ee4099 at {username='slave'} at 127.0.0.1:60820: session id = 9832bf960c714aff8c1dc4fc62ee4099-bc9d4e02a1e14b369f999b13dad44544
I20250626 01:53:20.296690 3928 tablet_copy_source_session.cc:215] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:20.297837 4783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9d4e02a1e14b369f999b13dad44544. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:20.299235 4785 ts_tablet_manager.cc:927] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: Initiating tablet copy from peer 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:20.300128 4785 tablet_copy_client.cc:323] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:40297
I20250626 01:53:20.300860 3854 tablet_copy_service.cc:140] P 311ee4b8be9f47a8a3de56840e44425a: Received BeginTabletCopySession request for tablet 71684e4b73c24f57b67e052454de9097 from peer 464c771a333746b6aab3fc7a84481415 ({username='slave'} at 127.0.0.1:57254)
I20250626 01:53:20.300982 3854 tablet_copy_service.cc:161] P 311ee4b8be9f47a8a3de56840e44425a: Beginning new tablet copy session on tablet 71684e4b73c24f57b67e052454de9097 from peer 464c771a333746b6aab3fc7a84481415 at {username='slave'} at 127.0.0.1:57254: session id = 464c771a333746b6aab3fc7a84481415-71684e4b73c24f57b67e052454de9097
I20250626 01:53:20.304020 3854 tablet_copy_source_session.cc:215] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:20.304656 4783 tablet_copy_client.cc:806] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:20.304963 4783 tablet_copy_client.cc:670] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:20.305420 4785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71684e4b73c24f57b67e052454de9097. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:20.306463 4784 tablet_copy_client.cc:1025] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:20.311647 4785 tablet_copy_client.cc:806] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:20.312026 4785 tablet_copy_client.cc:670] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:20.313750 4786 tablet_copy_client.cc:1025] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:20.406127 4787 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:20.406307 4787 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:20.407016 3918 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47160
I20250626 01:53:20.407215 3918 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:20.407313 3918 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:20.410005 3918 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:20.410620 3918 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 7 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297)
I20250626 01:53:20.411191 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:20.411238 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 7 candidate_status { last_received { term: 6 index: 31 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:20.411353 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:20.411394 3765 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 LEADER]: Stepping down as leader of term 6
I20250626 01:53:20.411612 3765 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:20.412120 3765 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 31, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:20.412452 3765 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:20.413916 3840 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 7.
I20250626 01:53:20.414165 3765 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 7.
I20250626 01:53:20.414472 3884 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:53:20.414755 4778 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 01:53:20.414882 4778 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:20.415112 4778 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 6.31, Last appended by leader: 31, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:20.418154 3667 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 6 to 7, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 7 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:20.478401 4789 ts_tablet_manager.cc:927] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: Initiating tablet copy from peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493)
I20250626 01:53:20.478986 4789 tablet_copy_client.cc:323] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.70:43493
I20250626 01:53:20.479583 4322 tablet_copy_service.cc:140] P 9832bf960c714aff8c1dc4fc62ee4099: Received BeginTabletCopySession request for tablet 96fb20902f87437487c5f36d71de0b9b from peer fb97ea56eeb04a3d9bf983411d630ee7 ({username='slave'} at 127.0.0.1:48670)
I20250626 01:53:20.479934 4322 tablet_copy_service.cc:161] P 9832bf960c714aff8c1dc4fc62ee4099: Beginning new tablet copy session on tablet 96fb20902f87437487c5f36d71de0b9b from peer fb97ea56eeb04a3d9bf983411d630ee7 at {username='slave'} at 127.0.0.1:48670: session id = fb97ea56eeb04a3d9bf983411d630ee7-96fb20902f87437487c5f36d71de0b9b
I20250626 01:53:20.482890 4775 raft_consensus.cc:991] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:20.482903 4322 tablet_copy_source_session.cc:215] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:20.483069 4775 raft_consensus.cc:1079] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:20.483556 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33212
I20250626 01:53:20.483731 3840 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:20.483937 3840 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:20.484192 4789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fb20902f87437487c5f36d71de0b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:20.486747 3840 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:20.487991 4163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
I20250626 01:53:20.488092 3840 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 6 election: Requested vote from peers 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233), fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:20.488195 4163 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:20.487977 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:20.488399 3918 raft_consensus.cc:3053] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 01:53:20.488469 3918 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:20.488713 3918 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:20.489140 3918 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:20.489833 4789 tablet_copy_client.cc:806] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:20.490036 4789 tablet_copy_client.cc:670] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:20.490906 4163 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 6.
I20250626 01:53:20.491312 3810 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, fb97ea56eeb04a3d9bf983411d630ee7; no voters:
I20250626 01:53:20.491572 4760 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Leader election won for term 6
I20250626 01:53:20.492040 4760 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:20.492307 4760 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:20.494225 3918 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 6.
I20250626 01:53:20.495030 4790 tablet_copy_client.cc:1025] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Injecting latency into file download: 2000
I20250626 01:53:20.494987 3667 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 5 to 6, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 6 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:20.502640 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:20.502856 3840 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Received request to transfer leadership
I20250626 01:53:20.507982 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:20.508170 3840 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Received request to transfer leadership
I20250626 01:53:20.509500 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:20.509651 3840 raft_consensus.cc:604] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Received request to transfer leadership
W20250626 01:53:20.510102 3715 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for bc9d4e02a1e14b369f999b13dad44544 already in progress
W20250626 01:53:20.510174 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for bc9d4e02a1e14b369f999b13dad44544 already in progress
I20250626 01:53:20.827164 3765 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 7 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20250626 01:53:20.827719 4778 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:20.843801 3840 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 31. Preceding OpId from leader: term: 7 index: 32. (index mismatch)
I20250626 01:53:20.844455 4775 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:20.975839 4162 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250626 01:53:20.976464 4793 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:20.979079 4760 raft_consensus.cc:991] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:20.979228 4760 raft_consensus.cc:1079] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Signalling peer fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:20.980526 4163 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544"
dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:54814
W20250626 01:53:20.981062 3807 consensus_peers.cc:487] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:20.981158 3918 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250626 01:53:20.982781 4791 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:20.983690 4163 raft_consensus.cc:491] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:20.983811 4163 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:20.986286 4163 raft_consensus.cc:513] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:20.987273 4163 leader_election.cc:290] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 7 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:20.987839 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:20.988039 3837 raft_consensus.cc:3053] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Stepping down as leader of term 6
I20250626 01:53:20.988094 3837 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:20.988339 3837 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:20.988729 3837 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:20.990530 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bc9d4e02a1e14b369f999b13dad44544" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 7 candidate_status { last_received { term: 6 index: 35 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:20.990717 3918 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:20.991565 3837 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 7.
I20250626 01:53:20.992110 4129 leader_election.cc:304] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, fb97ea56eeb04a3d9bf983411d630ee7; no voters:
I20250626 01:53:20.992352 4795 raft_consensus.cc:2802] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 FOLLOWER]: Leader election won for term 7
I20250626 01:53:20.992449 4795 raft_consensus.cc:695] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 LEADER]: Becoming Leader. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Running, Role: LEADER
I20250626 01:53:20.992697 4795 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 35, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:20.993314 3918 raft_consensus.cc:2466] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 7.
I20250626 01:53:20.995187 3667 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: term changed from 6 to 7, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68). New cstate: current_term: 7 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:21.391415 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250626 01:53:21.391950 4753 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
W20250626 01:53:21.395138 4126 consensus_peers.cc:487] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 -> Peer 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70:43493): Couldn't send request to peer 9832bf960c714aff8c1dc4fc62ee4099. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:21.395327 3918 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 7 index: 36. (index mismatch)
I20250626 01:53:21.396039 4753 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 01:53:21.972128 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:21.981371 4162 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:48692
I20250626 01:53:21.981567 4162 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:21.982537 3918 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "464c771a333746b6aab3fc7a84481415"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:21.982712 3918 raft_consensus.cc:604] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Received request to transfer leadership to TS 464c771a333746b6aab3fc7a84481415
I20250626 01:53:21.983093 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20250626 01:53:21.983319 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:22.186686 4813 raft_consensus.cc:991] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:22.186944 4812 raft_consensus.cc:1079] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:22.187556 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33268
I20250626 01:53:22.187740 3840 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:22.187848 3840 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:22.190455 3840 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.191087 3840 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 5 election: Requested vote from peers 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233), fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:22.191481 4163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
I20250626 01:53:22.191646 4163 raft_consensus.cc:3053] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 LEADER]: Stepping down as leader of term 4
I20250626 01:53:22.191710 4163 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Running, Role: LEADER
I20250626 01:53:22.191776 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 5 candidate_status { last_received { term: 4 index: 33 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:22.192001 3918 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:22.191993 4163 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.192262 4163 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 4 FOLLOWER]: Advancing to term 5
I20250626 01:53:22.194563 4163 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 5.
I20250626 01:53:22.194622 3918 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 5.
I20250626 01:53:22.195056 3810 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, fb97ea56eeb04a3d9bf983411d630ee7; no voters:
I20250626 01:53:22.195351 4804 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Leader election won for term 5
I20250626 01:53:22.195492 4804 raft_consensus.cc:695] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:22.195751 4804 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.199059 3660 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 4 to 5, leader changed from fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 5 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:22.208433 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:22.208631 3840 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Received request to transfer leadership
I20250626 01:53:22.213241 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:22.213433 3840 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Received request to transfer leadership
W20250626 01:53:22.213739 3654 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for eec015ad4dee46efb54559801ff9896b already in progress [suppressed 4 similar messages]
I20250626 01:53:22.214713 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:22.214883 3840 raft_consensus.cc:604] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Received request to transfer leadership
W20250626 01:53:22.215307 3715 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for eec015ad4dee46efb54559801ff9896b already in progress
W20250626 01:53:22.215368 3715 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for eec015ad4dee46efb54559801ff9896b already in progress
I20250626 01:53:22.286258 4779 tablet_copy_client.cc:538] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:22.289158 4779 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: Bootstrap starting.
I20250626 01:53:22.306861 4783 tablet_copy_client.cc:538] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:22.310138 4783 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap starting.
I20250626 01:53:22.314123 4785 tablet_copy_client.cc:538] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:22.318030 4785 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: Bootstrap starting.
I20250626 01:53:22.331346 4779 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: Bootstrap replayed 1/1 log segments. Stats: ops{read=33 overwritten=0 applied=33 ignored=0} inserts{seen=112 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:22.332039 4779 tablet_bootstrap.cc:492] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: Bootstrap complete.
I20250626 01:53:22.332468 4779 ts_tablet_manager.cc:1397] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: Time spent bootstrapping tablet: real 0.043s user 0.024s sys 0.018s
I20250626 01:53:22.333436 4779 raft_consensus.cc:357] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.333793 4779 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Initialized, Role: LEARNER
I20250626 01:53:22.334050 4779 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 33, Last appended: 4.33, Last appended by leader: 33, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.335424 4779 ts_tablet_manager.cc:1428] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250626 01:53:22.336745 4173 tablet_copy_service.cc:342] P fb97ea56eeb04a3d9bf983411d630ee7: Request end of tablet copy session 464c771a333746b6aab3fc7a84481415-eec015ad4dee46efb54559801ff9896b received from {username='slave'} at 127.0.0.1:48690
I20250626 01:53:22.336869 4173 tablet_copy_service.cc:434] P fb97ea56eeb04a3d9bf983411d630ee7: ending tablet copy session 464c771a333746b6aab3fc7a84481415-eec015ad4dee46efb54559801ff9896b on tablet eec015ad4dee46efb54559801ff9896b with peer 464c771a333746b6aab3fc7a84481415
I20250626 01:53:22.363404 4783 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:22.364068 4783 tablet_bootstrap.cc:492] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: Bootstrap complete.
I20250626 01:53:22.364351 4783 ts_tablet_manager.cc:1397] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: Time spent bootstrapping tablet: real 0.054s user 0.041s sys 0.011s
I20250626 01:53:22.365226 4783 raft_consensus.cc:357] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:22.365446 4783 raft_consensus.cc:738] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: 9832bf960c714aff8c1dc4fc62ee4099, State: Initialized, Role: LEARNER
I20250626 01:53:22.365682 4783 consensus_queue.cc:260] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true } }
I20250626 01:53:22.366840 4783 ts_tablet_manager.cc:1428] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:53:22.367537 3928 tablet_copy_service.cc:342] P 001b6ec9d9bc46c79e57a4915c44c925: Request end of tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-bc9d4e02a1e14b369f999b13dad44544 received from {username='slave'} at 127.0.0.1:60820
I20250626 01:53:22.367663 3928 tablet_copy_service.cc:434] P 001b6ec9d9bc46c79e57a4915c44c925: ending tablet copy session 9832bf960c714aff8c1dc4fc62ee4099-bc9d4e02a1e14b369f999b13dad44544 on tablet bc9d4e02a1e14b369f999b13dad44544 with peer 9832bf960c714aff8c1dc4fc62ee4099
I20250626 01:53:22.375869 4785 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=96 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:22.376410 4785 tablet_bootstrap.cc:492] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: Bootstrap complete.
I20250626 01:53:22.376678 4785 ts_tablet_manager.cc:1397] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: Time spent bootstrapping tablet: real 0.059s user 0.038s sys 0.014s
I20250626 01:53:22.377386 4785 raft_consensus.cc:357] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.377607 4785 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Initialized, Role: LEARNER
I20250626 01:53:22.377787 4785 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.378276 4785 ts_tablet_manager.cc:1428] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:53:22.378907 3854 tablet_copy_service.cc:342] P 311ee4b8be9f47a8a3de56840e44425a: Request end of tablet copy session 464c771a333746b6aab3fc7a84481415-71684e4b73c24f57b67e052454de9097 received from {username='slave'} at 127.0.0.1:57254
I20250626 01:53:22.379030 3854 tablet_copy_service.cc:434] P 311ee4b8be9f47a8a3de56840e44425a: ending tablet copy session 464c771a333746b6aab3fc7a84481415-71684e4b73c24f57b67e052454de9097 on tablet 71684e4b73c24f57b67e052454de9097 with peer 464c771a333746b6aab3fc7a84481415
I20250626 01:53:22.426735 4809 raft_consensus.cc:991] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 464c771a333746b6aab3fc7a84481415 to start an election
I20250626 01:53:22.426936 4825 raft_consensus.cc:1079] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Signalling peer 464c771a333746b6aab3fc7a84481415 to start an election
I20250626 01:53:22.427480 4237 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694"
dest_uuid: "464c771a333746b6aab3fc7a84481415"
from {username='slave'} at 127.0.0.1:47250
I20250626 01:53:22.427665 4237 raft_consensus.cc:491] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:22.427759 4237 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:22.430290 4237 raft_consensus.cc:513] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:22.431041 4237 leader_election.cc:290] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [CANDIDATE]: Term 6 election: Requested vote from peers 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233), 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305)
I20250626 01:53:22.431360 3918 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "464c771a333746b6aab3fc7a84481415" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:22.431411 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f43f00d4508b48278aeea9fb37615694" candidate_uuid: "464c771a333746b6aab3fc7a84481415" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:22.431533 3918 raft_consensus.cc:3053] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Stepping down as leader of term 5
I20250626 01:53:22.431602 3765 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:22.431614 3918 raft_consensus.cc:738] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:22.431974 3918 consensus_queue.cc:260] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:22.432248 3918 raft_consensus.cc:3058] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:22.433705 3765 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464c771a333746b6aab3fc7a84481415 in term 6.
I20250626 01:53:22.433928 3918 raft_consensus.cc:2466] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 464c771a333746b6aab3fc7a84481415 in term 6.
I20250626 01:53:22.434150 4204 leader_election.cc:304] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464c771a333746b6aab3fc7a84481415, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:22.434401 4745 raft_consensus.cc:2802] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 01:53:22.434755 4745 raft_consensus.cc:695] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 6 LEADER]: Becoming Leader. State: Replica: 464c771a333746b6aab3fc7a84481415, State: Running, Role: LEADER
I20250626 01:53:22.435022 4745 consensus_queue.cc:237] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:22.444923 3667 catalog_manager.cc:5582] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 reported cstate change: term changed from 5 to 6, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 464c771a333746b6aab3fc7a84481415 (127.0.103.69). New cstate: current_term: 6 leader_uuid: "464c771a333746b6aab3fc7a84481415" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:22.446974 4310 raft_consensus.cc:3058] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 5 LEARNER]: Advancing to term 7
I20250626 01:53:22.449923 4310 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 LEARNER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250626 01:53:22.450548 4812 consensus_queue.cc:1037] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:22.453509 4812 raft_consensus.cc:1062] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: attempting to promote NON_VOTER 9832bf960c714aff8c1dc4fc62ee4099 to VOTER
I20250626 01:53:22.454296 4812 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 7.36, Last appended by leader: 35, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:22.456764 3918 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
I20250626 01:53:22.457093 3840 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
I20250626 01:53:22.457466 4311 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 LEARNER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 7 index: 36. Preceding OpId from leader: term: 7 index: 37. (index mismatch)
I20250626 01:53:22.457693 4813 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 01:53:22.458142 4813 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 01:53:22.458473 4813 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 01:53:22.467006 4813 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 LEADER]: Committing config change with OpId 7.37: config changed from index 34 to 37, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.470479 4311 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 FOLLOWER]: Committing config change with OpId 7.37: config changed from index 34 to 37, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.473498 3837 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Committing config change with OpId 7.37: config changed from index 34 to 37, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.477639 3667 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: config changed from index 34 to 37, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 7 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.479574 3918 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Committing config change with OpId 7.37: config changed from index 34 to 37, 9832bf960c714aff8c1dc4fc62ee4099 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.486501 4163 consensus_queue.cc:237] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 37, Committed index: 37, Last appended: 7.37, Last appended by leader: 35, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } }
I20250626 01:53:22.488744 4311 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 7 index: 37. Preceding OpId from leader: term: 7 index: 38. (index mismatch)
I20250626 01:53:22.488890 3918 raft_consensus.cc:1273] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 7 index: 37. Preceding OpId from leader: term: 7 index: 38. (index mismatch)
I20250626 01:53:22.489495 4831 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20250626 01:53:22.489835 4812 consensus_queue.cc:1035] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 38, Last known committed idx: 37, Time since last communication: 0.000s
I20250626 01:53:22.492396 4831 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 LEADER]: Committing config change with OpId 7.38: config changed from index 37 to 38, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.493101 4311 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 FOLLOWER]: Committing config change with OpId 7.38: config changed from index 37 to 38, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.493185 3918 raft_consensus.cc:2953] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Committing config change with OpId 7.38: config changed from index 37 to 38, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.495340 4789 tablet_copy_client.cc:538] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:22.498942 4789 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap starting.
I20250626 01:53:22.498910 3659 catalog_manager.cc:5582] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: config changed from index 37 to 38, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New cstate: current_term: 7 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 38 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } }
I20250626 01:53:22.502903 3714 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet bc9d4e02a1e14b369f999b13dad44544 with cas_config_opid_index 37: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:22.503167 3826 tablet_service.cc:1515] Processing DeleteTablet for tablet bc9d4e02a1e14b369f999b13dad44544 with delete_type TABLET_DATA_TOMBSTONED (TS 311ee4b8be9f47a8a3de56840e44425a not found in new config with opid_index 38) from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:22.516358 4834 tablet_replica.cc:331] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:22.516640 4834 raft_consensus.cc:2241] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:22.517174 4834 raft_consensus.cc:2270] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:22.518806 4834 ts_tablet_manager.cc:1905] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:22.526087 4834 ts_tablet_manager.cc:1918] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.37
I20250626 01:53:22.526264 4834 log.cc:1199] T bc9d4e02a1e14b369f999b13dad44544 P 311ee4b8be9f47a8a3de56840e44425a: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/wals/bc9d4e02a1e14b369f999b13dad44544
I20250626 01:53:22.527247 3654 catalog_manager.cc:4928] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): tablet bc9d4e02a1e14b369f999b13dad44544 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:22.543637 4789 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:22.544152 4789 tablet_bootstrap.cc:492] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: Bootstrap complete.
I20250626 01:53:22.544422 4789 ts_tablet_manager.cc:1397] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: Time spent bootstrapping tablet: real 0.046s user 0.031s sys 0.012s
I20250626 01:53:22.545127 4789 raft_consensus.cc:357] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:22.545318 4789 raft_consensus.cc:738] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Initialized, Role: LEARNER
I20250626 01:53:22.545526 4789 consensus_queue.cc:260] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: true } }
I20250626 01:53:22.546021 4789 ts_tablet_manager.cc:1428] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:53:22.546672 4322 tablet_copy_service.cc:342] P 9832bf960c714aff8c1dc4fc62ee4099: Request end of tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-96fb20902f87437487c5f36d71de0b9b received from {username='slave'} at 127.0.0.1:48670
I20250626 01:53:22.546819 4322 tablet_copy_service.cc:434] P 9832bf960c714aff8c1dc4fc62ee4099: ending tablet copy session fb97ea56eeb04a3d9bf983411d630ee7-96fb20902f87437487c5f36d71de0b9b on tablet 96fb20902f87437487c5f36d71de0b9b with peer fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:22.623550 3918 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250626 01:53:22.624135 4804 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:22.626690 4791 raft_consensus.cc:991] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:22.626928 4804 raft_consensus.cc:1079] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:22.628101 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:22.630450 4237 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 4 LEARNER]: Advancing to term 5
I20250626 01:53:22.632274 4163 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250626 01:53:22.633001 4791 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:22.636168 4237 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 5 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 4 index: 33. Preceding OpId from leader: term: 5 index: 34. (index mismatch)
I20250626 01:53:22.636950 4791 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:22.640692 3916 raft_consensus.cc:491] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:22.640846 3916 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:22.646001 3916 raft_consensus.cc:513] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.647521 3837 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:22.647711 3837 raft_consensus.cc:3053] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Stepping down as leader of term 5
I20250626 01:53:22.647783 3837 raft_consensus.cc:738] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:22.647776 4162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "eec015ad4dee46efb54559801ff9896b" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 6 candidate_status { last_received { term: 5 index: 34 } } ignore_live_leader: true dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
I20250626 01:53:22.648083 3837 consensus_queue.cc:260] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.648334 3837 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:22.649178 3916 leader_election.cc:290] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 6 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397)
I20250626 01:53:22.650905 3837 raft_consensus.cc:2466] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 6.
I20250626 01:53:22.651808 3884 leader_election.cc:304] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters: fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:22.652398 4825 raft_consensus.cc:2802] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Leader election won for term 6
I20250626 01:53:22.652546 4825 raft_consensus.cc:695] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:22.652877 4825 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } }
I20250626 01:53:22.656586 3659 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 5 to 6, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 6 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:22.671208 4238 raft_consensus.cc:1215] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.29->[1.30-1.30] Dedup: 1.30->[]
I20250626 01:53:22.695278 4163 raft_consensus.cc:1215] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20250626 01:53:22.951733 3916 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20250626 01:53:22.952538 4745 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:22.957530 3765 raft_consensus.cc:1273] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Refusing update from remote peer 464c771a333746b6aab3fc7a84481415: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20250626 01:53:22.959563 4845 consensus_queue.cc:1035] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:23.074927 3840 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250626 01:53:23.075662 4809 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:23.078559 4238 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 5 LEARNER]: Advancing to term 6
I20250626 01:53:23.080763 4163 raft_consensus.cc:3058] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 5 FOLLOWER]: Advancing to term 6
I20250626 01:53:23.081920 4238 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 LEARNER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250626 01:53:23.082450 4809 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:23.083175 4163 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 5 index: 34. Preceding OpId from leader: term: 6 index: 35. (index mismatch)
I20250626 01:53:23.084162 4809 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:23.086889 4791 raft_consensus.cc:1062] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: attempting to promote NON_VOTER 464c771a333746b6aab3fc7a84481415 to VOTER
I20250626 01:53:23.087682 4791 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:23.091164 4849 raft_consensus.cc:1062] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: attempting to promote NON_VOTER 464c771a333746b6aab3fc7a84481415 to VOTER
I20250626 01:53:23.091995 4849 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 6.35, Last appended by leader: 34, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:23.093547 4163 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 01:53:23.093684 3765 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 01:53:23.094084 4238 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 LEARNER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 1 index: 31. (index mismatch)
I20250626 01:53:23.094235 4837 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:23.094622 4237 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 LEARNER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20250626 01:53:23.095065 4804 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:23.095937 4849 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 01:53:23.096673 3837 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20250626 01:53:23.098178 4825 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 01:53:23.095160 4163 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 35. Preceding OpId from leader: term: 6 index: 36. (index mismatch)
I20250626 01:53:23.100682 4825 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 01:53:23.101771 4809 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Committing config change with OpId 6.36: config changed from index 33 to 36, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.102326 3840 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Committing config change with OpId 6.36: config changed from index 33 to 36, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.108987 4804 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:23.109297 3659 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 33 to 36, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 6 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.130194 3916 consensus_queue.cc:237] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 6.36, Last appended by leader: 34, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:23.130987 4237 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Committing config change with OpId 6.36: config changed from index 33 to 36, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.134847 4161 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Committing config change with OpId 6.36: config changed from index 33 to 36, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 36 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } attrs { replace: true } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.135170 4236 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 36. Preceding OpId from leader: term: 6 index: 37. (index mismatch)
I20250626 01:53:23.135902 4825 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 01:53:23.137219 4804 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.137410 4163 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.140015 4162 raft_consensus.cc:1273] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 6 index: 36. Preceding OpId from leader: term: 6 index: 37. (index mismatch)
I20250626 01:53:23.141412 3765 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.140699 4238 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Committing config change with OpId 1.31: config changed from index 30 to 31, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.143184 3659 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 30 to 31, 464c771a333746b6aab3fc7a84481415 (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } attrs { replace: true } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.146790 4811 raft_consensus.cc:1062] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: attempting to promote NON_VOTER fb97ea56eeb04a3d9bf983411d630ee7 to VOTER
I20250626 01:53:23.147584 4811 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:23.148610 4825 consensus_queue.cc:1035] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20250626 01:53:23.149828 3840 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 1.31, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:23.151099 4857 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Committing config change with OpId 6.37: config changed from index 36 to 37, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.152716 4236 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250626 01:53:23.152961 4163 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 1 index: 31. Preceding OpId from leader: term: 1 index: 32. (index mismatch)
I20250626 01:53:23.153378 4859 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:23.153858 4859 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:23.154210 4236 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Committing config change with OpId 6.37: config changed from index 36 to 37, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.152201 4162 raft_consensus.cc:2953] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Committing config change with OpId 6.37: config changed from index 36 to 37, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.157825 3653 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet eec015ad4dee46efb54559801ff9896b with cas_config_opid_index 36: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:23.159456 4851 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.159684 3660 catalog_manager.cc:5582] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 36 to 37, VOTER 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) evicted. New cstate: current_term: 6 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.162148 4163 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEARNER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:23.163985 4855 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:23.164896 3654 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 71684e4b73c24f57b67e052454de9097 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:23.165216 3826 tablet_service.cc:1515] Processing DeleteTablet for tablet eec015ad4dee46efb54559801ff9896b with delete_type TABLET_DATA_TOMBSTONED (TS 311ee4b8be9f47a8a3de56840e44425a not found in new config with opid_index 37) from {username='slave'} at 127.0.0.1:33206
I20250626 01:53:23.165333 4236 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.166673 4862 tablet_replica.cc:331] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:23.166848 4862 raft_consensus.cc:2241] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:23.167446 4862 raft_consensus.cc:2270] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:23.167539 4162 raft_consensus.cc:2953] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Committing config change with OpId 1.32: config changed from index 31 to 32, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } } }
I20250626 01:53:23.169504 3840 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:23.171104 4811 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:23.168406 4862 ts_tablet_manager.cc:1905] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:23.167539 3659 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: config changed from index 31 to 32, VOTER 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:23.169504 3916 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20250626 01:53:23.173460 4855 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20250626 01:53:23.176800 3754 tablet_service.cc:1515] Processing DeleteTablet for tablet 71684e4b73c24f57b67e052454de9097 with delete_type TABLET_DATA_TOMBSTONED (TS 7e5284db6ec74f4a992cbd71b1dfc53f not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:41770
I20250626 01:53:23.187798 4865 tablet_replica.cc:331] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:23.189301 4865 raft_consensus.cc:2241] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:23.190876 4865 raft_consensus.cc:2270] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:23.191865 4862 ts_tablet_manager.cc:1918] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 6.36
I20250626 01:53:23.192080 4862 log.cc:1199] T eec015ad4dee46efb54559801ff9896b P 311ee4b8be9f47a8a3de56840e44425a: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-1-root/wals/eec015ad4dee46efb54559801ff9896b
I20250626 01:53:23.193223 4865 ts_tablet_manager.cc:1905] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:23.193987 4855 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.194639 4163 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.195276 3916 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.193511 3654 catalog_manager.cc:4928] TS 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): tablet eec015ad4dee46efb54559801ff9896b (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:23.201072 3660 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: config changed from index 30 to 31, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9832bf960c714aff8c1dc4fc62ee4099" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.201839 3840 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } attrs { replace: true } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.206738 4310 consensus_queue.cc:237] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } }
I20250626 01:53:23.209439 4163 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:23.210086 4855 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:23.212275 4861 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.212800 3837 raft_consensus.cc:1273] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer 9832bf960c714aff8c1dc4fc62ee4099: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20250626 01:53:23.213586 4855 consensus_queue.cc:1035] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20250626 01:53:23.216033 3709 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 96fb20902f87437487c5f36d71de0b9b with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:23.215688 3840 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.217264 4865 ts_tablet_manager.cc:1918] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.31
I20250626 01:53:23.217403 4865 log.cc:1199] T 71684e4b73c24f57b67e052454de9097 P 7e5284db6ec74f4a992cbd71b1dfc53f: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-0-root/wals/71684e4b73c24f57b67e052454de9097
I20250626 01:53:23.218219 3654 catalog_manager.cc:4928] TS 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): tablet 71684e4b73c24f57b67e052454de9097 (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:23.220446 4163 raft_consensus.cc:2953] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } }
I20250626 01:53:23.220736 3665 catalog_manager.cc:5582] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 reported cstate change: config changed from index 31 to 32, VOTER 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "9832bf960c714aff8c1dc4fc62ee4099" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9832bf960c714aff8c1dc4fc62ee4099" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 43493 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:53:23.224385 3904 tablet_service.cc:1515] Processing DeleteTablet for tablet 96fb20902f87437487c5f36d71de0b9b with delete_type TABLET_DATA_TOMBSTONED (TS 001b6ec9d9bc46c79e57a4915c44c925 not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:47144
I20250626 01:53:23.225399 4867 tablet_replica.cc:331] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:23.225745 4867 raft_consensus.cc:2241] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:23.226069 4867 raft_consensus.cc:2270] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:23.227227 4867 ts_tablet_manager.cc:1905] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:23.232234 4867 ts_tablet_manager.cc:1918] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20250626 01:53:23.232537 4867 log.cc:1199] T 96fb20902f87437487c5f36d71de0b9b P 001b6ec9d9bc46c79e57a4915c44c925: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0/minicluster-data/ts-2-root/wals/96fb20902f87437487c5f36d71de0b9b
I20250626 01:53:23.233500 3653 catalog_manager.cc:4928] TS 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233): tablet 96fb20902f87437487c5f36d71de0b9b (table test-workload [id=e97905cd9ed7422a85e4e45b4fe903ad]) successfully deleted
I20250626 01:53:23.983762 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:23.986419 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:23.986663 3916 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:23.987185 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:23.987423 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:24.273249 4825 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:24.273383 4825 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:24.273979 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33212
I20250626 01:53:24.274165 3840 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:24.274250 3840 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:24.276185 3840 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:24.276872 3840 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 7 election: Requested vote from peers 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:24.277344 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:24.277344 4236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:24.277597 3916 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Stepping down as leader of term 6
I20250626 01:53:24.277598 4236 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:24.277736 3916 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:24.278041 3916 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:24.278409 3916 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Advancing to term 7
I20250626 01:53:24.280727 4236 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 7.
I20250626 01:53:24.280997 3916 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 7.
I20250626 01:53:24.281375 3810 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 464c771a333746b6aab3fc7a84481415; no voters:
I20250626 01:53:24.281704 4858 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Leader election won for term 7
I20250626 01:53:24.282122 4858 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:24.282424 4858 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:24.285225 3659 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 6 to 7, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 7 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:24.706991 4236 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 7 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20250626 01:53:24.707628 4858 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:24.711644 3916 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20250626 01:53:24.712621 4869 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20250626 01:53:25.988204 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:25.991056 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:25.991318 3840 raft_consensus.cc:604] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 LEADER]: Received request to transfer leadership to TS 001b6ec9d9bc46c79e57a4915c44c925
I20250626 01:53:25.991868 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:25.992125 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:26.309888 4882 raft_consensus.cc:991] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:26.310074 4882 raft_consensus.cc:1079] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 LEADER]: Signalling peer 001b6ec9d9bc46c79e57a4915c44c925 to start an election
I20250626 01:53:26.310755 3916 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
from {username='slave'} at 127.0.0.1:47158
I20250626 01:53:26.310954 3916 raft_consensus.cc:491] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:26.311064 3916 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Advancing to term 8
I20250626 01:53:26.313074 3916 raft_consensus.cc:513] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:26.313656 3916 leader_election.cc:290] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 8 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:26.314152 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 8 candidate_status { last_received { term: 7 index: 35 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:26.314149 4236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "34f950fcb4ee4d5aad0d3f06ed2081af" candidate_uuid: "001b6ec9d9bc46c79e57a4915c44c925" candidate_term: 8 candidate_status { last_received { term: 7 index: 35 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:26.314330 3840 raft_consensus.cc:3053] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 LEADER]: Stepping down as leader of term 7
I20250626 01:53:26.314356 4236 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 7 FOLLOWER]: Advancing to term 8
I20250626 01:53:26.314399 3840 raft_consensus.cc:738] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:26.314652 3840 consensus_queue.cc:260] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 7.35, Last appended by leader: 35, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:26.314927 3840 raft_consensus.cc:3058] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Advancing to term 8
I20250626 01:53:26.316395 4236 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 8.
I20250626 01:53:26.316852 3884 leader_election.cc:304] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 464c771a333746b6aab3fc7a84481415; no voters:
I20250626 01:53:26.317126 4825 raft_consensus.cc:2802] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 FOLLOWER]: Leader election won for term 8
I20250626 01:53:26.317183 3840 raft_consensus.cc:2466] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 001b6ec9d9bc46c79e57a4915c44c925 in term 8.
I20250626 01:53:26.317515 4825 raft_consensus.cc:695] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 LEADER]: Becoming Leader. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:26.317798 4825 consensus_queue.cc:237] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 35, Committed index: 35, Last appended: 7.35, Last appended by leader: 35, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:26.320407 3659 catalog_manager.cc:5582] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 reported cstate change: term changed from 7 to 8, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67). New cstate: current_term: 8 leader_uuid: "001b6ec9d9bc46c79e57a4915c44c925" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:26.826261 4236 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 8 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 7 index: 35. Preceding OpId from leader: term: 8 index: 36. (index mismatch)
I20250626 01:53:26.826987 4825 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 01:53:26.829092 3840 raft_consensus.cc:1273] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Refusing update from remote peer 001b6ec9d9bc46c79e57a4915c44c925: Log matching property violated. Preceding OpId in replica: term: 7 index: 35. Preceding OpId from leader: term: 8 index: 36. (index mismatch)
I20250626 01:53:26.829943 4825 consensus_queue.cc:1035] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 36, Last known committed idx: 35, Time since last communication: 0.000s
I20250626 01:53:27.992589 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:27.995352 3916 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
mode: GRACEFUL
new_leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:47172
I20250626 01:53:27.995585 3916 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 LEADER]: Received request to transfer leadership to TS 7e5284db6ec74f4a992cbd71b1dfc53f
I20250626 01:53:27.996155 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:27.996348 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:28.314062 4898 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: : Instructing follower 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:28.314201 4898 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 LEADER]: Signalling peer 7e5284db6ec74f4a992cbd71b1dfc53f to start an election
I20250626 01:53:28.314898 3765 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
from {username='slave'} at 127.0.0.1:41786
I20250626 01:53:28.315079 3765 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:28.315182 3765 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 7 FOLLOWER]: Advancing to term 8
I20250626 01:53:28.317175 3765 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:28.317778 3765 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 8 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:28.318312 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:28.318313 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" candidate_term: 8 candidate_status { last_received { term: 7 index: 32 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:28.318506 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 7 FOLLOWER]: Advancing to term 8
I20250626 01:53:28.318490 3916 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 LEADER]: Stepping down as leader of term 7
I20250626 01:53:28.318670 3916 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 001b6ec9d9bc46c79e57a4915c44c925, State: Running, Role: LEADER
I20250626 01:53:28.318913 3916 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 7.32, Last appended by leader: 32, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:28.319183 3916 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Advancing to term 8
I20250626 01:53:28.320740 3840 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 8.
I20250626 01:53:28.321267 3735 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 7e5284db6ec74f4a992cbd71b1dfc53f; no voters:
I20250626 01:53:28.321358 3916 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5284db6ec74f4a992cbd71b1dfc53f in term 8.
I20250626 01:53:28.321550 4787 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 FOLLOWER]: Leader election won for term 8
I20250626 01:53:28.321976 4787 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 LEADER]: Becoming Leader. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:28.322255 4787 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 7.32, Last appended by leader: 32, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:28.324967 3665 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f reported cstate change: term changed from 7 to 8, leader changed from 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67) to 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65). New cstate: current_term: 8 leader_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:28.847373 3840 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 7 index: 32. Preceding OpId from leader: term: 8 index: 33. (index mismatch)
I20250626 01:53:28.848018 4787 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:28.852778 3916 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 FOLLOWER]: Refusing update from remote peer 7e5284db6ec74f4a992cbd71b1dfc53f: Log matching property violated. Preceding OpId in replica: term: 7 index: 32. Preceding OpId from leader: term: 8 index: 33. (index mismatch)
I20250626 01:53:28.853616 4787 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:29.996781 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:29.999812 3765 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:40602
I20250626 01:53:30.000111 3765 raft_consensus.cc:604] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:30.000648 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:30.000882 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:30.507923 4915 raft_consensus.cc:991] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:30.508082 4915 raft_consensus.cc:1079] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:30.508759 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "346b250db608442d84f96d2c2e603775"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33226
I20250626 01:53:30.508966 3840 raft_consensus.cc:491] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:30.509081 3840 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Advancing to term 9
I20250626 01:53:30.511087 3840 raft_consensus.cc:513] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:30.511854 3840 leader_election.cc:290] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 9 election: Requested vote from peers 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305), 001b6ec9d9bc46c79e57a4915c44c925 (127.0.103.67:35233)
I20250626 01:53:30.512411 3916 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 9 candidate_status { last_received { term: 8 index: 33 } } ignore_live_leader: true dest_uuid: "001b6ec9d9bc46c79e57a4915c44c925"
I20250626 01:53:30.512434 3765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "346b250db608442d84f96d2c2e603775" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 9 candidate_status { last_received { term: 8 index: 33 } } ignore_live_leader: true dest_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f"
I20250626 01:53:30.512619 3916 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 FOLLOWER]: Advancing to term 9
I20250626 01:53:30.512639 3765 raft_consensus.cc:3053] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 LEADER]: Stepping down as leader of term 8
I20250626 01:53:30.512753 3765 raft_consensus.cc:738] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 7e5284db6ec74f4a992cbd71b1dfc53f, State: Running, Role: LEADER
I20250626 01:53:30.513036 3765 consensus_queue.cc:260] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 8.33, Last appended by leader: 33, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:30.513327 3765 raft_consensus.cc:3058] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 8 FOLLOWER]: Advancing to term 9
I20250626 01:53:30.514751 3916 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 9.
I20250626 01:53:30.515233 3765 raft_consensus.cc:2466] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 9.
I20250626 01:53:30.515259 3809 leader_election.cc:304] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 001b6ec9d9bc46c79e57a4915c44c925, 311ee4b8be9f47a8a3de56840e44425a; no voters:
I20250626 01:53:30.515596 4882 raft_consensus.cc:2802] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 9 FOLLOWER]: Leader election won for term 9
I20250626 01:53:30.516002 4882 raft_consensus.cc:695] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 9 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:30.516300 4882 consensus_queue.cc:237] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 8.33, Last appended by leader: 33, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } }
I20250626 01:53:30.519248 3659 catalog_manager.cc:5582] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 8 to 9, leader changed from 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 9 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:30.957455 3765 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 9 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 8 index: 33. Preceding OpId from leader: term: 9 index: 34. (index mismatch)
I20250626 01:53:30.958422 4882 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e5284db6ec74f4a992cbd71b1dfc53f" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:30.963207 3916 raft_consensus.cc:1273] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 9 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 8 index: 33. Preceding OpId from leader: term: 9 index: 34. (index mismatch)
I20250626 01:53:30.965118 4925 consensus_queue.cc:1035] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "001b6ec9d9bc46c79e57a4915c44c925" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:32.001340 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:32.004246 3840 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "71684e4b73c24f57b67e052454de9097"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
mode: GRACEFUL
new_leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:57242
I20250626 01:53:32.004491 3840 raft_consensus.cc:604] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Received request to transfer leadership to TS fb97ea56eeb04a3d9bf983411d630ee7
I20250626 01:53:32.005017 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:32.005236 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:32.223846 4917 raft_consensus.cc:991] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: : Instructing follower fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:32.224117 4917 raft_consensus.cc:1079] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Signalling peer fb97ea56eeb04a3d9bf983411d630ee7 to start an election
I20250626 01:53:32.224725 4163 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "71684e4b73c24f57b67e052454de9097"
dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
from {username='slave'} at 127.0.0.1:54814
I20250626 01:53:32.224908 4163 raft_consensus.cc:491] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:32.225009 4163 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:32.226931 4163 raft_consensus.cc:513] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:32.227648 4163 leader_election.cc:290] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 2 election: Requested vote from peers 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:32.228147 3840 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
I20250626 01:53:32.228209 4236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:32.228307 3840 raft_consensus.cc:3053] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:32.228394 3840 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:32.228431 4236 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:32.228689 3840 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:32.229004 3840 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:32.230813 4236 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 2.
I20250626 01:53:32.230965 3840 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb97ea56eeb04a3d9bf983411d630ee7 in term 2.
I20250626 01:53:32.231408 4129 leader_election.cc:304] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 464c771a333746b6aab3fc7a84481415, fb97ea56eeb04a3d9bf983411d630ee7; no voters:
I20250626 01:53:32.231717 4831 raft_consensus.cc:2802] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:32.232141 4831 raft_consensus.cc:695] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Becoming Leader. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Running, Role: LEADER
I20250626 01:53:32.232409 4831 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 1.32, Last appended by leader: 32, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:32.235188 3665 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 reported cstate change: term changed from 1 to 2, leader changed from 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66) to fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68). New cstate: current_term: 2 leader_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:32.630381 4236 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 01:53:32.631065 4831 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:32.636801 3840 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Refusing update from remote peer fb97ea56eeb04a3d9bf983411d630ee7: Log matching property violated. Preceding OpId in replica: term: 1 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20250626 01:53:32.637821 4930 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20250626 01:53:34.005774 3720 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:34.008818 4163 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "71684e4b73c24f57b67e052454de9097"
dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
mode: GRACEFUL
new_leader_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:48692
I20250626 01:53:34.009064 4163 raft_consensus.cc:604] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Received request to transfer leadership to TS 311ee4b8be9f47a8a3de56840e44425a
I20250626 01:53:34.009716 3720 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:34.009950 3720 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:34.026547 4930 raft_consensus.cc:991] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: : Instructing follower 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:34.026707 4930 raft_consensus.cc:1079] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Signalling peer 311ee4b8be9f47a8a3de56840e44425a to start an election
I20250626 01:53:34.027488 3840 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "71684e4b73c24f57b67e052454de9097"
dest_uuid: "311ee4b8be9f47a8a3de56840e44425a"
from {username='slave'} at 127.0.0.1:33268
I20250626 01:53:34.027702 3840 raft_consensus.cc:491] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:34.027827 3840 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:34.030470 3840 raft_consensus.cc:513] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:34.031252 3840 leader_election.cc:290] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 3 election: Requested vote from peers fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68:44397), 464c771a333746b6aab3fc7a84481415 (127.0.103.69:45827)
I20250626 01:53:34.031778 4236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: true dest_uuid: "464c771a333746b6aab3fc7a84481415"
I20250626 01:53:34.031785 4163 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71684e4b73c24f57b67e052454de9097" candidate_uuid: "311ee4b8be9f47a8a3de56840e44425a" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: true dest_uuid: "fb97ea56eeb04a3d9bf983411d630ee7"
I20250626 01:53:34.032060 4163 raft_consensus.cc:3053] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Stepping down as leader of term 2
I20250626 01:53:34.032060 4236 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:34.032191 4163 raft_consensus.cc:738] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fb97ea56eeb04a3d9bf983411d630ee7, State: Running, Role: LEADER
I20250626 01:53:34.032512 4163 consensus_queue.cc:260] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:34.032907 4163 raft_consensus.cc:3058] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Advancing to term 3
I20250626 01:53:34.035127 4236 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 3.
I20250626 01:53:34.035543 4163 raft_consensus.cc:2466] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 311ee4b8be9f47a8a3de56840e44425a in term 3.
I20250626 01:53:34.035779 3810 leader_election.cc:304] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 311ee4b8be9f47a8a3de56840e44425a, 464c771a333746b6aab3fc7a84481415; no voters:
I20250626 01:53:34.036150 4917 raft_consensus.cc:2802] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Leader election won for term 3
I20250626 01:53:34.036590 4917 raft_consensus.cc:695] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Becoming Leader. State: Replica: 311ee4b8be9f47a8a3de56840e44425a, State: Running, Role: LEADER
I20250626 01:53:34.037089 4917 consensus_queue.cc:237] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.33, Last appended by leader: 33, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } }
I20250626 01:53:34.040344 3659 catalog_manager.cc:5582] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a reported cstate change: term changed from 2 to 3, leader changed from fb97ea56eeb04a3d9bf983411d630ee7 (127.0.103.68) to 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66). New cstate: current_term: 3 leader_uuid: "311ee4b8be9f47a8a3de56840e44425a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "311ee4b8be9f47a8a3de56840e44425a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40297 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:34.444639 4163 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250626 01:53:34.445356 4917 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb97ea56eeb04a3d9bf983411d630ee7" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20250626 01:53:34.449815 4236 raft_consensus.cc:1273] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Refusing update from remote peer 311ee4b8be9f47a8a3de56840e44425a: Log matching property violated. Preceding OpId in replica: term: 2 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20250626 01:53:34.450476 4917 consensus_queue.cc:1035] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [LEADER]: Connected to new peer: Peer: permanent_uuid: "464c771a333746b6aab3fc7a84481415" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 45827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:195: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20250626 01:53:34.606350 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:53:34.618610 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:34.618908 413 tablet_replica.cc:331] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:34.619118 413 raft_consensus.cc:2241] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.619475 413 raft_consensus.cc:2270] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.620172 413 tablet_replica.cc:331] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:34.620296 413 raft_consensus.cc:2241] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.620504 413 raft_consensus.cc:2270] T 978023fb60024f26a213be77e43d8140 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.620970 413 tablet_replica.cc:331] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:34.621093 413 raft_consensus.cc:2241] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.621368 413 raft_consensus.cc:2270] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.621779 413 tablet_replica.cc:331] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:34.621893 413 raft_consensus.cc:2241] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.622015 413 raft_consensus.cc:2270] T f43f00d4508b48278aeea9fb37615694 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.622407 413 tablet_replica.cc:331] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:34.622517 413 raft_consensus.cc:2241] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.622777 413 raft_consensus.cc:2270] T 16e2c24dbd4f446caf8d09565cbac048 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.623204 413 tablet_replica.cc:331] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f: stopping tablet replica
I20250626 01:53:34.623312 413 raft_consensus.cc:2241] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 9 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.623422 413 raft_consensus.cc:2270] T 346b250db608442d84f96d2c2e603775 P 7e5284db6ec74f4a992cbd71b1dfc53f [term 9 FOLLOWER]: Raft consensus is shut down!
W20250626 01:53:34.640185 3810 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250626 01:53:34.641445 3810 consensus_peers.cc:487] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a -> Peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): Couldn't send request to peer 7e5284db6ec74f4a992cbd71b1dfc53f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:34.657963 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:53:34.663470 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:53:34.674336 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:34.674718 413 tablet_replica.cc:331] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:34.674909 413 raft_consensus.cc:2241] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.675086 413 raft_consensus.cc:2270] T 34f950fcb4ee4d5aad0d3f06ed2081af P 311ee4b8be9f47a8a3de56840e44425a [term 8 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.675673 413 tablet_replica.cc:331] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:34.675792 413 raft_consensus.cc:2241] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.676100 413 raft_consensus.cc:2270] T 978023fb60024f26a213be77e43d8140 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.676554 413 tablet_replica.cc:331] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:34.676664 413 raft_consensus.cc:2241] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 3 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.676865 413 raft_consensus.cc:2270] T 71684e4b73c24f57b67e052454de9097 P 311ee4b8be9f47a8a3de56840e44425a [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.677522 413 tablet_replica.cc:331] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:34.677627 413 raft_consensus.cc:2241] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.677827 413 raft_consensus.cc:2270] T 16e2c24dbd4f446caf8d09565cbac048 P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.678212 413 tablet_replica.cc:331] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:34.678305 413 raft_consensus.cc:2241] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.678426 413 raft_consensus.cc:2270] T 96fb20902f87437487c5f36d71de0b9b P 311ee4b8be9f47a8a3de56840e44425a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.678817 413 tablet_replica.cc:331] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a: stopping tablet replica
I20250626 01:53:34.678907 413 raft_consensus.cc:2241] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 9 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.679116 413 raft_consensus.cc:2270] T 346b250db608442d84f96d2c2e603775 P 311ee4b8be9f47a8a3de56840e44425a [term 9 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.686800 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
W20250626 01:53:34.704191 4204 consensus_peers.cc:487] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 -> Peer 7e5284db6ec74f4a992cbd71b1dfc53f (127.0.103.65:34305): Couldn't send request to peer 7e5284db6ec74f4a992cbd71b1dfc53f. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:34305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:34.709435 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:53:34.719949 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:34.720245 413 tablet_replica.cc:331] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:34.720471 413 raft_consensus.cc:2241] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.720762 413 raft_consensus.cc:2270] T e80ca9ec9a27407fa4e5ae542fdad35d P 001b6ec9d9bc46c79e57a4915c44c925 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.721418 413 tablet_replica.cc:331] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:34.721582 413 raft_consensus.cc:2241] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.721722 413 raft_consensus.cc:2270] T f43f00d4508b48278aeea9fb37615694 P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.722199 413 tablet_replica.cc:331] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:34.722335 413 raft_consensus.cc:2241] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 9 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.722501 413 raft_consensus.cc:2270] T 346b250db608442d84f96d2c2e603775 P 001b6ec9d9bc46c79e57a4915c44c925 [term 9 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.722959 413 tablet_replica.cc:331] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:34.723096 413 raft_consensus.cc:2241] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.723361 413 raft_consensus.cc:2270] T eec015ad4dee46efb54559801ff9896b P 001b6ec9d9bc46c79e57a4915c44c925 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.723783 413 tablet_replica.cc:331] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:34.723937 413 raft_consensus.cc:2241] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.724156 413 raft_consensus.cc:2270] T bc9d4e02a1e14b369f999b13dad44544 P 001b6ec9d9bc46c79e57a4915c44c925 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.724586 413 tablet_replica.cc:331] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925: stopping tablet replica
I20250626 01:53:34.724695 413 raft_consensus.cc:2241] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.724953 413 raft_consensus.cc:2270] T 34f950fcb4ee4d5aad0d3f06ed2081af P 001b6ec9d9bc46c79e57a4915c44c925 [term 8 FOLLOWER]: Raft consensus is shut down!
W20250626 01:53:34.738428 4278 consensus_peers.cc:487] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 -> Peer 311ee4b8be9f47a8a3de56840e44425a (127.0.103.66:40297): Couldn't send request to peer 311ee4b8be9f47a8a3de56840e44425a. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40297: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:34.741019 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:53:34.746425 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:53:34.756825 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:34.757084 413 tablet_replica.cc:331] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7: stopping tablet replica
I20250626 01:53:34.757258 413 raft_consensus.cc:2241] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.757480 413 raft_consensus.cc:2270] T 96fb20902f87437487c5f36d71de0b9b P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.758112 413 tablet_replica.cc:331] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7: stopping tablet replica
I20250626 01:53:34.758240 413 raft_consensus.cc:2241] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.758536 413 raft_consensus.cc:2270] T bc9d4e02a1e14b369f999b13dad44544 P fb97ea56eeb04a3d9bf983411d630ee7 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.759042 413 tablet_replica.cc:331] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7: stopping tablet replica
I20250626 01:53:34.759189 413 raft_consensus.cc:2241] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.759372 413 raft_consensus.cc:2270] T 978023fb60024f26a213be77e43d8140 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.759770 413 tablet_replica.cc:331] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7: stopping tablet replica
I20250626 01:53:34.759876 413 raft_consensus.cc:2241] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.760157 413 raft_consensus.cc:2270] T 0ce2bf1f3ecc44f7824389e2febfcb4f P fb97ea56eeb04a3d9bf983411d630ee7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.760542 413 tablet_replica.cc:331] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7: stopping tablet replica
I20250626 01:53:34.760638 413 raft_consensus.cc:2241] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.760757 413 raft_consensus.cc:2270] T eec015ad4dee46efb54559801ff9896b P fb97ea56eeb04a3d9bf983411d630ee7 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.761123 413 tablet_replica.cc:331] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7: stopping tablet replica
I20250626 01:53:34.761214 413 raft_consensus.cc:2241] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.761310 413 raft_consensus.cc:2270] T 71684e4b73c24f57b67e052454de9097 P fb97ea56eeb04a3d9bf983411d630ee7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.778163 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:53:34.797484 413 tablet_server.cc:178] TabletServer@127.0.103.69:0 shutting down...
I20250626 01:53:34.807420 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:34.807719 413 tablet_replica.cc:331] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415: stopping tablet replica
I20250626 01:53:34.807955 413 raft_consensus.cc:2241] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.808226 413 raft_consensus.cc:2270] T 71684e4b73c24f57b67e052454de9097 P 464c771a333746b6aab3fc7a84481415 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.809091 413 tablet_replica.cc:331] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415: stopping tablet replica
I20250626 01:53:34.809257 413 raft_consensus.cc:2241] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.809468 413 raft_consensus.cc:2270] T eec015ad4dee46efb54559801ff9896b P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.809978 413 tablet_replica.cc:331] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415: stopping tablet replica
I20250626 01:53:34.810122 413 raft_consensus.cc:2241] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.810300 413 raft_consensus.cc:2270] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.810742 413 tablet_replica.cc:331] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415: stopping tablet replica
I20250626 01:53:34.810878 413 raft_consensus.cc:2241] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 8 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.811046 413 raft_consensus.cc:2270] T 34f950fcb4ee4d5aad0d3f06ed2081af P 464c771a333746b6aab3fc7a84481415 [term 8 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.811473 413 tablet_replica.cc:331] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415: stopping tablet replica
I20250626 01:53:34.811601 413 raft_consensus.cc:2241] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.811877 413 raft_consensus.cc:2270] T e80ca9ec9a27407fa4e5ae542fdad35d P 464c771a333746b6aab3fc7a84481415 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.812354 413 tablet_replica.cc:331] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415: stopping tablet replica
I20250626 01:53:34.812484 413 raft_consensus.cc:2241] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 6 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.812716 413 raft_consensus.cc:2270] T f43f00d4508b48278aeea9fb37615694 P 464c771a333746b6aab3fc7a84481415 [term 6 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.817631 413 tablet_server.cc:195] TabletServer@127.0.103.69:0 shutdown complete.
I20250626 01:53:34.821652 413 tablet_server.cc:178] TabletServer@127.0.103.70:0 shutting down...
I20250626 01:53:34.832314 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:34.832640 413 tablet_replica.cc:331] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099: stopping tablet replica
I20250626 01:53:34.832854 413 raft_consensus.cc:2241] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.833123 413 raft_consensus.cc:2270] T 0ce2bf1f3ecc44f7824389e2febfcb4f P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.833758 413 tablet_replica.cc:331] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099: stopping tablet replica
I20250626 01:53:34.833887 413 raft_consensus.cc:2241] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.834115 413 raft_consensus.cc:2270] T e80ca9ec9a27407fa4e5ae542fdad35d P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.834596 413 tablet_replica.cc:331] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099: stopping tablet replica
I20250626 01:53:34.834726 413 raft_consensus.cc:2241] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.834923 413 raft_consensus.cc:2270] T bc9d4e02a1e14b369f999b13dad44544 P 9832bf960c714aff8c1dc4fc62ee4099 [term 7 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.835381 413 tablet_replica.cc:331] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099: stopping tablet replica
I20250626 01:53:34.835489 413 raft_consensus.cc:2241] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.835680 413 raft_consensus.cc:2270] T 20ad5196a1fa4032afaa62ccab7e3fd1 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.836175 413 tablet_replica.cc:331] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099: stopping tablet replica
I20250626 01:53:34.836282 413 raft_consensus.cc:2241] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:34.836458 413 raft_consensus.cc:2270] T 16e2c24dbd4f446caf8d09565cbac048 P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.836860 413 tablet_replica.cc:331] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099: stopping tablet replica
I20250626 01:53:34.836977 413 raft_consensus.cc:2241] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.837255 413 raft_consensus.cc:2270] T 96fb20902f87437487c5f36d71de0b9b P 9832bf960c714aff8c1dc4fc62ee4099 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.852494 413 tablet_server.cc:195] TabletServer@127.0.103.70:0 shutdown complete.
I20250626 01:53:34.856968 413 master.cc:561] Master@127.0.103.126:34867 shutting down...
I20250626 01:53:34.867304 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:34.867563 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:34.867642 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 618d88e4b2554811b16a814603e1da85: stopping tablet replica
I20250626 01:53:34.883044 413 master.cc:583] Master@127.0.103.126:34867 shutdown complete.
I20250626 01:53:34.894520 413 test_util.cc:183] -----------------------------------------------
I20250626 01:53:34.894613 413 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1750902684575999-413-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (37147 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250626 01:53:34.896682 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36431
I20250626 01:53:34.897228 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:34.899585 4960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:34.899698 4961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:34.905565 4963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:34.906049 413 server_base.cc:1048] running on GCE node
I20250626 01:53:34.907289 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:34.907442 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:34.907491 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902814907486 us; error 0 us; skew 500 ppm
I20250626 01:53:34.907653 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:34.913713 413 webserver.cc:469] Webserver started at http://127.0.103.126:37513/ using document root <none> and password file <none>
I20250626 01:53:34.914006 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:34.914095 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:34.914251 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:34.914948 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "482a6bc7237245cab186eb68576803b0"
format_stamp: "Formatted at 2025-06-26 01:53:34 on dist-test-slave-psm5"
I20250626 01:53:34.917825 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20250626 01:53:34.919811 4968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:34.920617 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250626 01:53:34.920763 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "482a6bc7237245cab186eb68576803b0"
format_stamp: "Formatted at 2025-06-26 01:53:34 on dist-test-slave-psm5"
I20250626 01:53:34.920894 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:34.940964 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:34.941524 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:34.955682 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36431
I20250626 01:53:34.955682 5019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36431 every 8 connection(s)
I20250626 01:53:34.957024 5020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:34.961711 5020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0: Bootstrap starting.
I20250626 01:53:34.963131 5020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:34.964882 5020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0: No bootstrap required, opened a new log
I20250626 01:53:34.965576 5020 raft_consensus.cc:357] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482a6bc7237245cab186eb68576803b0" member_type: VOTER }
I20250626 01:53:34.965703 5020 raft_consensus.cc:383] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:34.965751 5020 raft_consensus.cc:738] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482a6bc7237245cab186eb68576803b0, State: Initialized, Role: FOLLOWER
I20250626 01:53:34.965947 5020 consensus_queue.cc:260] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482a6bc7237245cab186eb68576803b0" member_type: VOTER }
I20250626 01:53:34.966053 5020 raft_consensus.cc:397] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:53:34.966110 5020 raft_consensus.cc:491] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:53:34.966181 5020 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:34.968042 5020 raft_consensus.cc:513] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482a6bc7237245cab186eb68576803b0" member_type: VOTER }
I20250626 01:53:34.968230 5020 leader_election.cc:304] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 482a6bc7237245cab186eb68576803b0; no voters:
I20250626 01:53:34.968662 5020 leader_election.cc:290] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:53:34.968835 5023 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:34.969337 5023 raft_consensus.cc:695] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 1 LEADER]: Becoming Leader. State: Replica: 482a6bc7237245cab186eb68576803b0, State: Running, Role: LEADER
I20250626 01:53:34.969592 5020 sys_catalog.cc:564] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:53:34.969574 5023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482a6bc7237245cab186eb68576803b0" member_type: VOTER }
I20250626 01:53:34.970726 5025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 482a6bc7237245cab186eb68576803b0. Latest consensus state: current_term: 1 leader_uuid: "482a6bc7237245cab186eb68576803b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482a6bc7237245cab186eb68576803b0" member_type: VOTER } }
I20250626 01:53:34.970904 5025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:53:34.971228 5024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "482a6bc7237245cab186eb68576803b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482a6bc7237245cab186eb68576803b0" member_type: VOTER } }
I20250626 01:53:34.971462 5024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [sys.catalog]: This master's current role is: LEADER
I20250626 01:53:34.972560 5030 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:53:34.974081 5030 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:53:34.977111 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:53:34.977414 5030 catalog_manager.cc:1349] Generated new cluster ID: f4f047c7b366435c9baa0f17c466fc9d
I20250626 01:53:34.977487 5030 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:53:34.990294 5030 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:53:34.991158 5030 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:53:35.001359 5030 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0: Generated new TSK 0
I20250626 01:53:35.001657 5030 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:53:35.010413 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:35.013852 5041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:35.014063 5042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:35.014714 5044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:35.014981 413 server_base.cc:1048] running on GCE node
I20250626 01:53:35.015233 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:35.015283 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:35.015326 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902815015322 us; error 0 us; skew 500 ppm
I20250626 01:53:35.015491 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:35.016999 413 webserver.cc:469] Webserver started at http://127.0.103.65:44341/ using document root <none> and password file <none>
I20250626 01:53:35.017238 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:35.017305 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:35.017438 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:35.018085 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5"
format_stamp: "Formatted at 2025-06-26 01:53:35 on dist-test-slave-psm5"
I20250626 01:53:35.020879 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:53:35.023557 5049 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.024119 413 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250626 01:53:35.024221 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5"
format_stamp: "Formatted at 2025-06-26 01:53:35 on dist-test-slave-psm5"
I20250626 01:53:35.024351 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:35.044147 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:35.044706 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:35.045286 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:35.046175 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:35.046233 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.046310 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:35.046353 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.060726 5111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41927 every 8 connection(s)
I20250626 01:53:35.060699 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41927
I20250626 01:53:35.062849 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:35.065991 5116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:35.066334 5117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:35.066686 5119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:35.067951 413 server_base.cc:1048] running on GCE node
I20250626 01:53:35.068262 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:35.068325 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:35.068380 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902815068376 us; error 0 us; skew 500 ppm
I20250626 01:53:35.068526 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:35.069540 5112 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36431
I20250626 01:53:35.069675 5112 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:35.070271 413 webserver.cc:469] Webserver started at http://127.0.103.66:41375/ using document root <none> and password file <none>
I20250626 01:53:35.070494 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:35.070358 5112 heartbeater.cc:507] Master 127.0.103.126:36431 requested a full tablet report, sending...
I20250626 01:53:35.070613 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:35.070766 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:35.071552 4985 ts_manager.cc:194] Registered new tserver with Master: 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65:41927)
I20250626 01:53:35.071607 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
format_stamp: "Formatted at 2025-06-26 01:53:35 on dist-test-slave-psm5"
I20250626 01:53:35.072649 4985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56964
I20250626 01:53:35.074615 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250626 01:53:35.076424 5124 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.076803 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:53:35.076905 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
format_stamp: "Formatted at 2025-06-26 01:53:35 on dist-test-slave-psm5"
I20250626 01:53:35.077018 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:35.086506 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:35.087098 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:35.087734 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:35.088662 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:35.088735 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.088820 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:35.088856 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.104249 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36191
I20250626 01:53:35.104274 5186 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36191 every 8 connection(s)
I20250626 01:53:35.105788 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:35.108598 5191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:35.110004 5193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:35.111358 5190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:35.112756 413 server_base.cc:1048] running on GCE node
I20250626 01:53:35.113029 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:35.113099 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:35.113095 5187 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36431
I20250626 01:53:35.113215 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902815113191 us; error 0 us; skew 500 ppm
I20250626 01:53:35.113279 5187 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:35.113380 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:35.113667 5187 heartbeater.cc:507] Master 127.0.103.126:36431 requested a full tablet report, sending...
I20250626 01:53:35.114600 4985 ts_manager.cc:194] Registered new tserver with Master: 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191)
I20250626 01:53:35.114917 413 webserver.cc:469] Webserver started at http://127.0.103.67:38067/ using document root <none> and password file <none>
I20250626 01:53:35.115128 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:35.115197 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:35.115314 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:35.115605 4985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56978
I20250626 01:53:35.116053 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "0295adb472d04f918860545c2773ca2e"
format_stamp: "Formatted at 2025-06-26 01:53:35 on dist-test-slave-psm5"
I20250626 01:53:35.118703 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250626 01:53:35.120283 5198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.120640 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:53:35.120738 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "0295adb472d04f918860545c2773ca2e"
format_stamp: "Formatted at 2025-06-26 01:53:35 on dist-test-slave-psm5"
I20250626 01:53:35.120844 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:35.128041 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:35.128553 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:35.129097 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:35.129910 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:35.129976 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.130065 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:35.130110 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:35.142674 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:33893
I20250626 01:53:35.142802 5260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:33893 every 8 connection(s)
I20250626 01:53:35.150599 5261 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36431
I20250626 01:53:35.150769 5261 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:35.151172 5261 heartbeater.cc:507] Master 127.0.103.126:36431 requested a full tablet report, sending...
I20250626 01:53:35.152097 4985 ts_manager.cc:194] Registered new tserver with Master: 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893)
I20250626 01:53:35.152220 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008227675s
I20250626 01:53:35.152884 4985 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56990
I20250626 01:53:36.074137 5112 heartbeater.cc:499] Master 127.0.103.126:36431 was elected leader, sending a full tablet report...
I20250626 01:53:36.117126 5187 heartbeater.cc:499] Master 127.0.103.126:36431 was elected leader, sending a full tablet report...
I20250626 01:53:36.154165 5261 heartbeater.cc:499] Master 127.0.103.126:36431 was elected leader, sending a full tablet report...
I20250626 01:53:36.180557 413 test_util.cc:276] Using random seed: -731575994
I20250626 01:53:36.196241 4985 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36594:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:53:36.197018 4985 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:53:36.219413 5077 tablet_service.cc:1468] Processing CreateTablet for tablet 0383d42e16bc4bdf9874ca20f0c08166 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 01:53:36.219413 5076 tablet_service.cc:1468] Processing CreateTablet for tablet 1581a1ced1744769b0c6970d5e539c64 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 01:53:36.220072 5077 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0383d42e16bc4bdf9874ca20f0c08166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.220194 5075 tablet_service.cc:1468] Processing CreateTablet for tablet 8b32b514cc2549928fb3a52a488e8498 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 01:53:36.220633 5075 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b32b514cc2549928fb3a52a488e8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.220675 5225 tablet_service.cc:1468] Processing CreateTablet for tablet 1581a1ced1744769b0c6970d5e539c64 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 01:53:36.221103 5225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1581a1ced1744769b0c6970d5e539c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.222334 5226 tablet_service.cc:1468] Processing CreateTablet for tablet 0383d42e16bc4bdf9874ca20f0c08166 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 01:53:36.222724 5226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0383d42e16bc4bdf9874ca20f0c08166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.224453 5076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1581a1ced1744769b0c6970d5e539c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.228502 5224 tablet_service.cc:1468] Processing CreateTablet for tablet 8b32b514cc2549928fb3a52a488e8498 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 01:53:36.228528 5151 tablet_service.cc:1468] Processing CreateTablet for tablet 1581a1ced1744769b0c6970d5e539c64 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250626 01:53:36.229782 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1581a1ced1744769b0c6970d5e539c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.229172 5150 tablet_service.cc:1468] Processing CreateTablet for tablet 8b32b514cc2549928fb3a52a488e8498 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250626 01:53:36.232720 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b32b514cc2549928fb3a52a488e8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.228528 5152 tablet_service.cc:1468] Processing CreateTablet for tablet 0383d42e16bc4bdf9874ca20f0c08166 (DEFAULT_TABLE table=test-workload [id=747abaf9bb004674a15f9c13814f9b53]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250626 01:53:36.236590 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0383d42e16bc4bdf9874ca20f0c08166. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.229171 5224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b32b514cc2549928fb3a52a488e8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:36.263110 5283 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Bootstrap starting.
I20250626 01:53:36.266552 5283 tablet_bootstrap.cc:654] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.270929 5281 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: Bootstrap starting.
I20250626 01:53:36.271847 5282 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Bootstrap starting.
I20250626 01:53:36.273975 5281 tablet_bootstrap.cc:654] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.273975 5282 tablet_bootstrap.cc:654] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.275301 5283 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: No bootstrap required, opened a new log
I20250626 01:53:36.275609 5283 ts_tablet_manager.cc:1397] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20250626 01:53:36.276865 5283 raft_consensus.cc:357] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.277094 5283 raft_consensus.cc:383] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.277181 5283 raft_consensus.cc:738] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d307b5bb994b0aa6eb5aa46e9f56bd, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.277432 5283 consensus_queue.cc:260] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.277783 5281 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: No bootstrap required, opened a new log
I20250626 01:53:36.277877 5282 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: No bootstrap required, opened a new log
I20250626 01:53:36.277949 5281 ts_tablet_manager.cc:1397] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250626 01:53:36.278008 5282 ts_tablet_manager.cc:1397] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250626 01:53:36.278254 5283 ts_tablet_manager.cc:1428] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:53:36.278491 5283 tablet_bootstrap.cc:492] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Bootstrap starting.
I20250626 01:53:36.278697 5281 raft_consensus.cc:357] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.278844 5281 raft_consensus.cc:383] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.278903 5281 raft_consensus.cc:738] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0295adb472d04f918860545c2773ca2e, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.278988 5282 raft_consensus.cc:357] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.279129 5282 raft_consensus.cc:383] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.279219 5282 raft_consensus.cc:738] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e6bf5a75fa4624ba5a671cfaf71cb5, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.279247 5281 consensus_queue.cc:260] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.279551 5282 consensus_queue.cc:260] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.280490 5283 tablet_bootstrap.cc:654] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.281438 5281 ts_tablet_manager.cc:1428] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250626 01:53:36.281844 5281 tablet_bootstrap.cc:492] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e: Bootstrap starting.
I20250626 01:53:36.282684 5283 tablet_bootstrap.cc:492] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd: No bootstrap required, opened a new log
I20250626 01:53:36.282897 5283 ts_tablet_manager.cc:1397] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250626 01:53:36.283911 5281 tablet_bootstrap.cc:654] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.283799 5283 raft_consensus.cc:357] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.284179 5283 raft_consensus.cc:383] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.284304 5283 raft_consensus.cc:738] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d307b5bb994b0aa6eb5aa46e9f56bd, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.284584 5283 consensus_queue.cc:260] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.285281 5283 ts_tablet_manager.cc:1428] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:53:36.285611 5283 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Bootstrap starting.
I20250626 01:53:36.287817 5283 tablet_bootstrap.cc:654] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.288657 5282 ts_tablet_manager.cc:1428] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20250626 01:53:36.288959 5282 tablet_bootstrap.cc:492] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Bootstrap starting.
I20250626 01:53:36.289538 5281 tablet_bootstrap.cc:492] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e: No bootstrap required, opened a new log
I20250626 01:53:36.289687 5281 ts_tablet_manager.cc:1397] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20250626 01:53:36.290362 5281 raft_consensus.cc:357] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.290537 5281 raft_consensus.cc:383] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.290601 5281 raft_consensus.cc:738] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0295adb472d04f918860545c2773ca2e, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.290808 5281 consensus_queue.cc:260] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.291209 5281 ts_tablet_manager.cc:1428] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 01:53:36.291400 5281 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: Bootstrap starting.
I20250626 01:53:36.293221 5281 tablet_bootstrap.cc:654] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.298138 5282 tablet_bootstrap.cc:654] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.299715 5281 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: No bootstrap required, opened a new log
I20250626 01:53:36.299770 5283 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd: No bootstrap required, opened a new log
I20250626 01:53:36.299991 5281 ts_tablet_manager.cc:1397] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20250626 01:53:36.300016 5283 ts_tablet_manager.cc:1397] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.008s
I20250626 01:53:36.300388 5282 tablet_bootstrap.cc:492] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: No bootstrap required, opened a new log
I20250626 01:53:36.300511 5282 ts_tablet_manager.cc:1397] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.001s
I20250626 01:53:36.300899 5283 raft_consensus.cc:357] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.300884 5281 raft_consensus.cc:357] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.301086 5283 raft_consensus.cc:383] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.301200 5281 raft_consensus.cc:383] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.301144 5282 raft_consensus.cc:357] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.301261 5281 raft_consensus.cc:738] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0295adb472d04f918860545c2773ca2e, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.301229 5283 raft_consensus.cc:738] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d307b5bb994b0aa6eb5aa46e9f56bd, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.301339 5282 raft_consensus.cc:383] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.301412 5282 raft_consensus.cc:738] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e6bf5a75fa4624ba5a671cfaf71cb5, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.301556 5283 consensus_queue.cc:260] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.301549 5281 consensus_queue.cc:260] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.301630 5282 consensus_queue.cc:260] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.302068 5283 ts_tablet_manager.cc:1428] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:53:36.302668 5281 ts_tablet_manager.cc:1428] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:53:36.304462 5282 ts_tablet_manager.cc:1428] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250626 01:53:36.304867 5282 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Bootstrap starting.
I20250626 01:53:36.306793 5282 tablet_bootstrap.cc:654] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:36.308732 5282 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5: No bootstrap required, opened a new log
I20250626 01:53:36.308907 5282 ts_tablet_manager.cc:1397] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20250626 01:53:36.309559 5282 raft_consensus.cc:357] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.309762 5282 raft_consensus.cc:383] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:36.309840 5282 raft_consensus.cc:738] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e6bf5a75fa4624ba5a671cfaf71cb5, State: Initialized, Role: FOLLOWER
I20250626 01:53:36.310060 5282 consensus_queue.cc:260] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.310465 5282 ts_tablet_manager.cc:1428] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250626 01:53:36.313225 5113 tablet.cc:2378] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:53:36.319394 5288 raft_consensus.cc:491] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:36.319603 5288 raft_consensus.cc:513] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.320384 5288 leader_election.cc:290] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65:41927), 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191)
I20250626 01:53:36.327030 5087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1581a1ced1744769b0c6970d5e539c64" candidate_uuid: "0295adb472d04f918860545c2773ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" is_pre_election: true
I20250626 01:53:36.327147 5162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1581a1ced1744769b0c6970d5e539c64" candidate_uuid: "0295adb472d04f918860545c2773ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" is_pre_election: true
I20250626 01:53:36.327265 5087 raft_consensus.cc:2466] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0295adb472d04f918860545c2773ca2e in term 0.
I20250626 01:53:36.327378 5162 raft_consensus.cc:2466] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0295adb472d04f918860545c2773ca2e in term 0.
I20250626 01:53:36.327726 5201 leader_election.cc:304] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0295adb472d04f918860545c2773ca2e, 42e6bf5a75fa4624ba5a671cfaf71cb5; no voters:
I20250626 01:53:36.328012 5288 raft_consensus.cc:2802] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:36.328087 5288 raft_consensus.cc:491] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:36.328163 5288 raft_consensus.cc:3058] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.330081 5288 raft_consensus.cc:513] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.330607 5288 leader_election.cc:290] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [CANDIDATE]: Term 1 election: Requested vote from peers 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65:41927), 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191)
I20250626 01:53:36.331103 5087 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1581a1ced1744769b0c6970d5e539c64" candidate_uuid: "0295adb472d04f918860545c2773ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5"
I20250626 01:53:36.331163 5162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1581a1ced1744769b0c6970d5e539c64" candidate_uuid: "0295adb472d04f918860545c2773ca2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
I20250626 01:53:36.331247 5087 raft_consensus.cc:3058] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.331315 5162 raft_consensus.cc:3058] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.333559 5162 raft_consensus.cc:2466] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0295adb472d04f918860545c2773ca2e in term 1.
I20250626 01:53:36.333559 5087 raft_consensus.cc:2466] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0295adb472d04f918860545c2773ca2e in term 1.
I20250626 01:53:36.333977 5201 leader_election.cc:304] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d307b5bb994b0aa6eb5aa46e9f56bd, 0295adb472d04f918860545c2773ca2e; no voters:
I20250626 01:53:36.334230 5288 raft_consensus.cc:2802] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:36.334643 5288 raft_consensus.cc:695] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 LEADER]: Becoming Leader. State: Replica: 0295adb472d04f918860545c2773ca2e, State: Running, Role: LEADER
I20250626 01:53:36.334873 5288 consensus_queue.cc:237] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.337414 4984 catalog_manager.cc:5582] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0295adb472d04f918860545c2773ca2e (127.0.103.67). New cstate: current_term: 1 leader_uuid: "0295adb472d04f918860545c2773ca2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:36.340921 5289 raft_consensus.cc:491] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:36.341089 5289 raft_consensus.cc:513] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.341845 5289 leader_election.cc:290] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893), 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191)
I20250626 01:53:36.347260 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0295adb472d04f918860545c2773ca2e" is_pre_election: true
I20250626 01:53:36.347260 5162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" is_pre_election: true
I20250626 01:53:36.347570 5162 raft_consensus.cc:2466] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 0.
I20250626 01:53:36.347571 5236 raft_consensus.cc:2466] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 0.
I20250626 01:53:36.348001 5052 leader_election.cc:304] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d307b5bb994b0aa6eb5aa46e9f56bd, 42e6bf5a75fa4624ba5a671cfaf71cb5; no voters:
I20250626 01:53:36.348263 5289 raft_consensus.cc:2802] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:36.348333 5289 raft_consensus.cc:491] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:36.348382 5289 raft_consensus.cc:3058] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.350440 5302 raft_consensus.cc:491] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:36.350482 5289 raft_consensus.cc:513] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.350570 5302 raft_consensus.cc:513] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.351523 5302 leader_election.cc:290] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191), 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893)
I20250626 01:53:36.351838 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0295adb472d04f918860545c2773ca2e"
I20250626 01:53:36.352058 5236 raft_consensus.cc:3058] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.352190 5235 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b32b514cc2549928fb3a52a488e8498" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0295adb472d04f918860545c2773ca2e" is_pre_election: true
I20250626 01:53:36.352335 5235 raft_consensus.cc:2466] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 0.
I20250626 01:53:36.352355 5161 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b32b514cc2549928fb3a52a488e8498" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" is_pre_election: true
I20250626 01:53:36.352573 5161 raft_consensus.cc:2466] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 0.
I20250626 01:53:36.352712 5053 leader_election.cc:304] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0295adb472d04f918860545c2773ca2e, 42e6bf5a75fa4624ba5a671cfaf71cb5; no voters:
I20250626 01:53:36.352838 5162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
I20250626 01:53:36.352977 5162 raft_consensus.cc:3058] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.352991 5302 raft_consensus.cc:2802] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:36.353084 5302 raft_consensus.cc:491] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:36.353150 5302 raft_consensus.cc:3058] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.354207 5236 raft_consensus.cc:2466] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 1.
I20250626 01:53:36.351583 5289 leader_election.cc:290] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 election: Requested vote from peers 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893), 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191)
I20250626 01:53:36.354739 5053 leader_election.cc:304] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0295adb472d04f918860545c2773ca2e, 42e6bf5a75fa4624ba5a671cfaf71cb5; no voters:
I20250626 01:53:36.354962 5289 raft_consensus.cc:2802] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:36.355405 5289 raft_consensus.cc:695] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Becoming Leader. State: Replica: 42e6bf5a75fa4624ba5a671cfaf71cb5, State: Running, Role: LEADER
I20250626 01:53:36.355715 5289 consensus_queue.cc:237] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:36.355794 5302 raft_consensus.cc:513] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.355780 5162 raft_consensus.cc:2466] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 1.
I20250626 01:53:36.356952 5302 leader_election.cc:290] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 election: Requested vote from peers 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191), 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893)
I20250626 01:53:36.357306 5162 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b32b514cc2549928fb3a52a488e8498" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
I20250626 01:53:36.357336 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8b32b514cc2549928fb3a52a488e8498" candidate_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0295adb472d04f918860545c2773ca2e"
I20250626 01:53:36.357455 5162 raft_consensus.cc:3058] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.357478 5236 raft_consensus.cc:3058] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:36.359283 5236 raft_consensus.cc:2466] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 1.
I20250626 01:53:36.359284 5162 raft_consensus.cc:2466] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e6bf5a75fa4624ba5a671cfaf71cb5 in term 1.
I20250626 01:53:36.359647 5053 leader_election.cc:304] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0295adb472d04f918860545c2773ca2e, 42e6bf5a75fa4624ba5a671cfaf71cb5; no voters:
I20250626 01:53:36.359963 5302 raft_consensus.cc:2802] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:36.360102 5302 raft_consensus.cc:695] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Becoming Leader. State: Replica: 42e6bf5a75fa4624ba5a671cfaf71cb5, State: Running, Role: LEADER
I20250626 01:53:36.360324 5302 consensus_queue.cc:237] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } }
I20250626 01:53:36.361055 4984 catalog_manager.cc:5582] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:36.365320 4984 catalog_manager.cc:5582] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:36.377807 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:36.380826 5306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:36.381073 5309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:36.381201 413 server_base.cc:1048] running on GCE node
W20250626 01:53:36.380965 5307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:36.381763 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:36.381831 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:36.381857 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902816381854 us; error 0 us; skew 500 ppm
I20250626 01:53:36.382022 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:36.383531 413 webserver.cc:469] Webserver started at http://127.0.103.68:35311/ using document root <none> and password file <none>
I20250626 01:53:36.383790 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:36.383857 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:36.384007 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:36.384707 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "8739c33f64e94f4bb25ac4504138c5ae"
format_stamp: "Formatted at 2025-06-26 01:53:36 on dist-test-slave-psm5"
I20250626 01:53:36.387429 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:53:36.389236 5314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:36.389647 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:53:36.389762 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "8739c33f64e94f4bb25ac4504138c5ae"
format_stamp: "Formatted at 2025-06-26 01:53:36 on dist-test-slave-psm5"
I20250626 01:53:36.389879 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:36.409428 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:36.410046 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:36.410662 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:36.411669 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:36.411736 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:36.411793 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:36.411824 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:36.421736 5288 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:53:36.425779 5288 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250626 01:53:36.425909 5288 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250626 01:53:36.430428 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:44883
I20250626 01:53:36.430438 5378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:44883 every 8 connection(s)
I20250626 01:53:36.437215 5379 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36431
I20250626 01:53:36.437427 5379 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:36.437899 5379 heartbeater.cc:507] Master 127.0.103.126:36431 requested a full tablet report, sending...
I20250626 01:53:36.438709 4984 ts_manager.cc:194] Registered new tserver with Master: 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68:44883)
I20250626 01:53:36.438776 5302 consensus_queue.cc:1035] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:36.439630 4984 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36600
I20250626 01:53:36.455790 5303 consensus_queue.cc:1035] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:36.457140 5288 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:36.478133 5289 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:36.480195 5289 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:53:36.976990 5038 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:36.985715 5087 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166"
dest_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5"
mode: GRACEFUL
new_leader_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
from {username='slave'} at 127.0.0.1:49284
I20250626 01:53:36.985915 5087 raft_consensus.cc:604] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Received request to transfer leadership to TS 00d307b5bb994b0aa6eb5aa46e9f56bd
I20250626 01:53:36.986546 5038 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250626 01:53:36.986605 5086 consensus_queue.cc:237] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } }
I20250626 01:53:36.986959 5038 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:36.989317 5236 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:53:36.989602 5162 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:53:36.990213 5289 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:36.991206 5302 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:36.995981 5162 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
I20250626 01:53:36.995817 5235 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
I20250626 01:53:37.001369 4982 catalog_manager.cc:5582] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd reported cstate change: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
W20250626 01:53:37.003551 5051 consensus_peers.cc:487] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 -> Peer 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68:44883): Couldn't send request to peer 8739c33f64e94f4bb25ac4504138c5ae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b32b514cc2549928fb3a52a488e8498. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:37.008092 5303 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
I20250626 01:53:37.018122 5235 consensus_queue.cc:237] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } }
I20250626 01:53:37.020447 5162 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:53:37.020479 5086 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:53:37.021085 5390 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:37.021411 5298 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:37.024734 5086 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
I20250626 01:53:37.024740 5162 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
W20250626 01:53:37.027706 5200 consensus_peers.cc:487] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e -> Peer 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68:44883): Couldn't send request to peer 8739c33f64e94f4bb25ac4504138c5ae. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1581a1ced1744769b0c6970d5e539c64. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:37.062356 4983 catalog_manager.cc:5582] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd reported cstate change: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "0295adb472d04f918860545c2773ca2e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
I20250626 01:53:37.066056 5390 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } } }
I20250626 01:53:37.071261 5404 ts_tablet_manager.cc:927] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: Initiating tablet copy from peer 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65:41927)
I20250626 01:53:37.072079 5404 tablet_copy_client.cc:323] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:41927
I20250626 01:53:37.079195 5097 tablet_copy_service.cc:140] P 42e6bf5a75fa4624ba5a671cfaf71cb5: Received BeginTabletCopySession request for tablet 8b32b514cc2549928fb3a52a488e8498 from peer 8739c33f64e94f4bb25ac4504138c5ae ({username='slave'} at 127.0.0.1:49296)
I20250626 01:53:37.079392 5097 tablet_copy_service.cc:161] P 42e6bf5a75fa4624ba5a671cfaf71cb5: Beginning new tablet copy session on tablet 8b32b514cc2549928fb3a52a488e8498 from peer 8739c33f64e94f4bb25ac4504138c5ae at {username='slave'} at 127.0.0.1:49296: session id = 8739c33f64e94f4bb25ac4504138c5ae-8b32b514cc2549928fb3a52a488e8498
I20250626 01:53:37.082005 5097 tablet_copy_source_session.cc:215] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:37.083217 5404 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b32b514cc2549928fb3a52a488e8498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:37.084264 5302 raft_consensus.cc:991] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: : Instructing follower 00d307b5bb994b0aa6eb5aa46e9f56bd to start an election
I20250626 01:53:37.084440 5303 raft_consensus.cc:1079] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Signalling peer 00d307b5bb994b0aa6eb5aa46e9f56bd to start an election
I20250626 01:53:37.085052 5162 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166"
dest_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd"
from {username='slave'} at 127.0.0.1:58890
I20250626 01:53:37.085245 5162 raft_consensus.cc:491] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250626 01:53:37.085383 5162 raft_consensus.cc:3058] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:37.088207 5404 tablet_copy_client.cc:806] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:37.088160 5162 raft_consensus.cc:513] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:37.088505 5404 tablet_copy_client.cc:670] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:37.089381 5162 leader_election.cc:290] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [CANDIDATE]: Term 2 election: Requested vote from peers 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65:41927), 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893)
I20250626 01:53:37.090128 5404 tablet_copy_client.cc:538] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:37.094861 5404 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: Bootstrap starting.
I20250626 01:53:37.097375 5086 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166" candidate_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5"
I20250626 01:53:37.097608 5086 raft_consensus.cc:3053] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Stepping down as leader of term 1
I20250626 01:53:37.097689 5086 raft_consensus.cc:738] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 42e6bf5a75fa4624ba5a671cfaf71cb5, State: Running, Role: LEADER
I20250626 01:53:37.097971 5086 consensus_queue.cc:260] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:37.098318 5086 raft_consensus.cc:3058] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:37.101493 5086 raft_consensus.cc:2466] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d307b5bb994b0aa6eb5aa46e9f56bd in term 2.
I20250626 01:53:37.101976 5127 leader_election.cc:304] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00d307b5bb994b0aa6eb5aa46e9f56bd, 42e6bf5a75fa4624ba5a671cfaf71cb5; no voters:
I20250626 01:53:37.102464 5397 raft_consensus.cc:2802] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:53:37.103397 5397 raft_consensus.cc:695] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 2 LEADER]: Becoming Leader. State: Replica: 00d307b5bb994b0aa6eb5aa46e9f56bd, State: Running, Role: LEADER
I20250626 01:53:37.103818 5397 consensus_queue.cc:237] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } }
I20250626 01:53:37.110733 5236 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0383d42e16bc4bdf9874ca20f0c08166" candidate_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0295adb472d04f918860545c2773ca2e"
I20250626 01:53:37.110960 5236 raft_consensus.cc:3058] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:53:37.111219 4982 catalog_manager.cc:5582] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd reported cstate change: term changed from 1 to 2, leader changed from 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65) to 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66). New cstate: current_term: 2 leader_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:37.112756 5404 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:37.113193 5404 tablet_bootstrap.cc:492] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: Bootstrap complete.
I20250626 01:53:37.113421 5404 ts_tablet_manager.cc:1397] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.001s
I20250626 01:53:37.113766 5236 raft_consensus.cc:2466] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00d307b5bb994b0aa6eb5aa46e9f56bd in term 2.
I20250626 01:53:37.114212 5404 raft_consensus.cc:357] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } }
I20250626 01:53:37.114439 5404 raft_consensus.cc:738] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8739c33f64e94f4bb25ac4504138c5ae, State: Initialized, Role: LEARNER
I20250626 01:53:37.114638 5404 consensus_queue.cc:260] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } }
I20250626 01:53:37.116514 5409 ts_tablet_manager.cc:927] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: Initiating tablet copy from peer 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893)
I20250626 01:53:37.117513 5379 heartbeater.cc:499] Master 127.0.103.126:36431 was elected leader, sending a full tablet report...
I20250626 01:53:37.117568 5404 ts_tablet_manager.cc:1428] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250626 01:53:37.118185 5097 tablet_copy_service.cc:342] P 42e6bf5a75fa4624ba5a671cfaf71cb5: Request end of tablet copy session 8739c33f64e94f4bb25ac4504138c5ae-8b32b514cc2549928fb3a52a488e8498 received from {username='slave'} at 127.0.0.1:49296
I20250626 01:53:37.118289 5097 tablet_copy_service.cc:434] P 42e6bf5a75fa4624ba5a671cfaf71cb5: ending tablet copy session 8739c33f64e94f4bb25ac4504138c5ae-8b32b514cc2549928fb3a52a488e8498 on tablet 8b32b514cc2549928fb3a52a488e8498 with peer 8739c33f64e94f4bb25ac4504138c5ae
I20250626 01:53:37.124282 5409 tablet_copy_client.cc:323] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33893
I20250626 01:53:37.129891 5246 tablet_copy_service.cc:140] P 0295adb472d04f918860545c2773ca2e: Received BeginTabletCopySession request for tablet 1581a1ced1744769b0c6970d5e539c64 from peer 8739c33f64e94f4bb25ac4504138c5ae ({username='slave'} at 127.0.0.1:50988)
I20250626 01:53:37.130108 5246 tablet_copy_service.cc:161] P 0295adb472d04f918860545c2773ca2e: Beginning new tablet copy session on tablet 1581a1ced1744769b0c6970d5e539c64 from peer 8739c33f64e94f4bb25ac4504138c5ae at {username='slave'} at 127.0.0.1:50988: session id = 8739c33f64e94f4bb25ac4504138c5ae-1581a1ced1744769b0c6970d5e539c64
I20250626 01:53:37.132021 5246 tablet_copy_source_session.cc:215] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:53:37.133626 5409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1581a1ced1744769b0c6970d5e539c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:37.139179 5409 tablet_copy_client.cc:806] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Starting download of 0 data blocks...
I20250626 01:53:37.139495 5409 tablet_copy_client.cc:670] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Starting download of 1 WAL segments...
I20250626 01:53:37.141264 5409 tablet_copy_client.cc:538] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:53:37.143790 5409 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: Bootstrap starting.
I20250626 01:53:37.149214 5409 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:53:37.149591 5409 tablet_bootstrap.cc:492] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: Bootstrap complete.
I20250626 01:53:37.149863 5409 ts_tablet_manager.cc:1397] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20250626 01:53:37.150573 5409 raft_consensus.cc:357] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } }
I20250626 01:53:37.150735 5409 raft_consensus.cc:738] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8739c33f64e94f4bb25ac4504138c5ae, State: Initialized, Role: LEARNER
I20250626 01:53:37.150913 5409 consensus_queue.cc:260] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: true } }
I20250626 01:53:37.151351 5409 ts_tablet_manager.cc:1428] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:53:37.151997 5246 tablet_copy_service.cc:342] P 0295adb472d04f918860545c2773ca2e: Request end of tablet copy session 8739c33f64e94f4bb25ac4504138c5ae-1581a1ced1744769b0c6970d5e539c64 received from {username='slave'} at 127.0.0.1:50988
I20250626 01:53:37.152110 5246 tablet_copy_service.cc:434] P 0295adb472d04f918860545c2773ca2e: ending tablet copy session 8739c33f64e94f4bb25ac4504138c5ae-1581a1ced1744769b0c6970d5e539c64 on tablet 1581a1ced1744769b0c6970d5e539c64 with peer 8739c33f64e94f4bb25ac4504138c5ae
I20250626 01:53:37.175868 5352 raft_consensus.cc:1215] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 01:53:37.182423 5352 raft_consensus.cc:1215] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 01:53:37.204241 5236 raft_consensus.cc:1273] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 2 FOLLOWER]: Refusing update from remote peer 00d307b5bb994b0aa6eb5aa46e9f56bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:53:37.204804 5397 consensus_queue.cc:1035] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:37.221416 5086 raft_consensus.cc:1273] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 2 FOLLOWER]: Refusing update from remote peer 00d307b5bb994b0aa6eb5aa46e9f56bd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:53:37.221972 5417 consensus_queue.cc:1035] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:53:37.299510 5302 raft_consensus.cc:1062] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: attempting to promote NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae to VOTER
I20250626 01:53:37.300381 5302 consensus_queue.cc:237] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } }
I20250626 01:53:37.302083 5352 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:53:37.302083 5235 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:53:37.302589 5289 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:53:37.302858 5162 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:53:37.303361 5289 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:53:37.303548 5302 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:53:37.307391 5352 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.307983 5162 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.312893 4982 catalog_manager.cc:5582] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae reported cstate change: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.362102 5390 raft_consensus.cc:1062] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: attempting to promote NON_VOTER 8739c33f64e94f4bb25ac4504138c5ae to VOTER
I20250626 01:53:37.362848 5390 consensus_queue.cc:237] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } }
I20250626 01:53:37.364537 5162 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:53:37.364703 5086 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:53:37.364789 5352 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 LEARNER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:53:37.365278 5288 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:53:37.365581 5403 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:53:37.365859 5298 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:53:37.369236 5087 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.369769 5352 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.374382 4984 catalog_manager.cc:5582] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae reported cstate change: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0295adb472d04f918860545c2773ca2e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.387876 5235 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.399958 5289 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } attrs { replace: true } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.413396 5086 consensus_queue.cc:237] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } }
I20250626 01:53:37.415038 5352 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:53:37.415038 5162 raft_consensus.cc:1273] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Refusing update from remote peer 42e6bf5a75fa4624ba5a671cfaf71cb5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:53:37.415663 5302 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:53:37.415824 5421 consensus_queue.cc:1035] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:53:37.418741 5352 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0295adb472d04f918860545c2773ca2e (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.418685 5162 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0295adb472d04f918860545c2773ca2e (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.423642 5403 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.423836 4982 catalog_manager.cc:5582] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd reported cstate change: config changed from index 3 to 4, VOTER 0295adb472d04f918860545c2773ca2e (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.427697 5162 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8739c33f64e94f4bb25ac4504138c5ae (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } attrs { replace: true } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.430174 5235 consensus_queue.cc:237] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } }
I20250626 01:53:37.432341 5352 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:53:37.432364 5087 raft_consensus.cc:1273] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Refusing update from remote peer 0295adb472d04f918860545c2773ca2e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:53:37.433025 5288 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:53:37.433228 5390 consensus_queue.cc:1035] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:53:37.436183 5352 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.436525 5086 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.441730 4983 catalog_manager.cc:5582] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 reported cstate change: config changed from index 3 to 4, VOTER 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "0295adb472d04f918860545c2773ca2e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.458922 5225 tablet_service.cc:1515] Processing DeleteTablet for tablet 8b32b514cc2549928fb3a52a488e8498 with delete_type TABLET_DATA_TOMBSTONED (TS 0295adb472d04f918860545c2773ca2e not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50948
I20250626 01:53:37.459699 5422 tablet_replica.cc:331] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: stopping tablet replica
I20250626 01:53:37.459904 5422 raft_consensus.cc:2241] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:37.460211 5422 raft_consensus.cc:2270] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:37.461627 5422 ts_tablet_manager.cc:1905] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:37.466981 5422 ts_tablet_manager.cc:1918] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 01:53:37.467110 5422 log.cc:1199] T 8b32b514cc2549928fb3a52a488e8498 P 0295adb472d04f918860545c2773ca2e: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-2-root/wals/8b32b514cc2549928fb3a52a488e8498
I20250626 01:53:37.467855 4972 catalog_manager.cc:4928] TS 0295adb472d04f918860545c2773ca2e (127.0.103.67:33893): tablet 8b32b514cc2549928fb3a52a488e8498 (table test-workload [id=747abaf9bb004674a15f9c13814f9b53]) successfully deleted
I20250626 01:53:37.481370 5150 tablet_service.cc:1515] Processing DeleteTablet for tablet 1581a1ced1744769b0c6970d5e539c64 with delete_type TABLET_DATA_TOMBSTONED (TS 00d307b5bb994b0aa6eb5aa46e9f56bd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58874
I20250626 01:53:37.482065 5423 tablet_replica.cc:331] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: stopping tablet replica
I20250626 01:53:37.482362 5423 raft_consensus.cc:2241] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:37.482728 5423 raft_consensus.cc:2270] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:37.483575 5423 ts_tablet_manager.cc:1905] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:53:37.489912 5423 ts_tablet_manager.cc:1918] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250626 01:53:37.490075 5423 log.cc:1199] T 1581a1ced1744769b0c6970d5e539c64 P 00d307b5bb994b0aa6eb5aa46e9f56bd: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1750902684575999-413-0/minicluster-data/ts-1-root/wals/1581a1ced1744769b0c6970d5e539c64
I20250626 01:53:37.490825 4971 catalog_manager.cc:4928] TS 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66:36191): tablet 1581a1ced1744769b0c6970d5e539c64 (table test-workload [id=747abaf9bb004674a15f9c13814f9b53]) successfully deleted
I20250626 01:53:37.530843 5289 raft_consensus.cc:2953] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0295adb472d04f918860545c2773ca2e (127.0.103.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "00d307b5bb994b0aa6eb5aa46e9f56bd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36191 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.534257 4971 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8b32b514cc2549928fb3a52a488e8498 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:37.584952 5288 raft_consensus.cc:2953] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 00d307b5bb994b0aa6eb5aa46e9f56bd (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "42e6bf5a75fa4624ba5a671cfaf71cb5" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41927 } } peers { permanent_uuid: "0295adb472d04f918860545c2773ca2e" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33893 } } peers { permanent_uuid: "8739c33f64e94f4bb25ac4504138c5ae" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 44883 } attrs { promote: false } } }
I20250626 01:53:37.588178 4972 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 1581a1ced1744769b0c6970d5e539c64 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:53:38.987440 5038 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:38.988381 5038 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:38.988552 5038 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:40.989045 5038 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:40.989719 5038 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:40.989822 5038 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:42.990298 5038 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:42.991020 5038 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:42.991130 5038 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:44.991519 5038 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:44.992431 5038 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:44.992553 5038 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:45.495391 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:53:45.506462 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:45.506773 413 tablet_replica.cc:331] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5: stopping tablet replica
I20250626 01:53:45.506973 413 raft_consensus.cc:2241] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:45.507221 413 raft_consensus.cc:2270] T 1581a1ced1744769b0c6970d5e539c64 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.507834 413 tablet_replica.cc:331] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5: stopping tablet replica
I20250626 01:53:45.508013 413 raft_consensus.cc:2241] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:45.508319 413 raft_consensus.cc:2270] T 8b32b514cc2549928fb3a52a488e8498 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.508821 413 tablet_replica.cc:331] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5: stopping tablet replica
I20250626 01:53:45.508953 413 raft_consensus.cc:2241] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:45.509086 413 raft_consensus.cc:2270] T 0383d42e16bc4bdf9874ca20f0c08166 P 42e6bf5a75fa4624ba5a671cfaf71cb5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.527119 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:53:45.530593 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
W20250626 01:53:45.535344 5201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41927: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250626 01:53:45.539062 5201 consensus_peers.cc:487] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e -> Peer 42e6bf5a75fa4624ba5a671cfaf71cb5 (127.0.103.65:41927): Couldn't send request to peer 42e6bf5a75fa4624ba5a671cfaf71cb5. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:45.542418 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:45.542739 413 tablet_replica.cc:331] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd: stopping tablet replica
I20250626 01:53:45.542933 413 raft_consensus.cc:2241] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:45.543181 413 raft_consensus.cc:2270] T 8b32b514cc2549928fb3a52a488e8498 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.543744 413 tablet_replica.cc:331] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd: stopping tablet replica
I20250626 01:53:45.543854 413 raft_consensus.cc:2241] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:53:45.544142 413 raft_consensus.cc:2270] T 0383d42e16bc4bdf9874ca20f0c08166 P 00d307b5bb994b0aa6eb5aa46e9f56bd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.558480 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:53:45.562041 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:53:45.573462 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:45.573913 413 tablet_replica.cc:331] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e: stopping tablet replica
I20250626 01:53:45.574153 413 raft_consensus.cc:2241] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:45.574764 413 raft_consensus.cc:2270] T 1581a1ced1744769b0c6970d5e539c64 P 0295adb472d04f918860545c2773ca2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.575623 413 tablet_replica.cc:331] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e: stopping tablet replica
I20250626 01:53:45.575812 413 raft_consensus.cc:2241] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:45.576014 413 raft_consensus.cc:2270] T 0383d42e16bc4bdf9874ca20f0c08166 P 0295adb472d04f918860545c2773ca2e [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.592340 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:53:45.595875 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:53:45.606501 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:45.606768 413 tablet_replica.cc:331] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae: stopping tablet replica
I20250626 01:53:45.606966 413 raft_consensus.cc:2241] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:45.607206 413 raft_consensus.cc:2270] T 1581a1ced1744769b0c6970d5e539c64 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.607815 413 tablet_replica.cc:331] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae: stopping tablet replica
I20250626 01:53:45.607991 413 raft_consensus.cc:2241] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:45.608163 413 raft_consensus.cc:2270] T 8b32b514cc2549928fb3a52a488e8498 P 8739c33f64e94f4bb25ac4504138c5ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.622120 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:53:45.642989 413 master.cc:561] Master@127.0.103.126:36431 shutting down...
I20250626 01:53:45.652781 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:45.653075 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:45.653183 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 482a6bc7237245cab186eb68576803b0: stopping tablet replica
I20250626 01:53:45.667464 413 master.cc:583] Master@127.0.103.126:36431 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10784 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250626 01:53:45.681238 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36405
I20250626 01:53:45.681795 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:45.684495 5435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:45.684834 413 server_base.cc:1048] running on GCE node
W20250626 01:53:45.684868 5436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.684963 5438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:45.685319 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:45.685380 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:45.685405 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902825685401 us; error 0 us; skew 500 ppm
I20250626 01:53:45.685591 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:45.691170 413 webserver.cc:469] Webserver started at http://127.0.103.126:41295/ using document root <none> and password file <none>
I20250626 01:53:45.691465 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:45.691538 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:45.691672 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:45.692425 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "a19a270290fb44348ab21cff96a4f1e6"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.695422 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250626 01:53:45.697365 5443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.697778 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:53:45.697896 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "a19a270290fb44348ab21cff96a4f1e6"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.698021 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:45.718075 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:45.718644 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:45.735867 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36405
I20250626 01:53:45.735911 5494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36405 every 8 connection(s)
I20250626 01:53:45.737389 5495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:45.741767 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6: Bootstrap starting.
I20250626 01:53:45.743356 5495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:45.745391 5495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6: No bootstrap required, opened a new log
I20250626 01:53:45.746184 5495 raft_consensus.cc:357] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a270290fb44348ab21cff96a4f1e6" member_type: VOTER }
I20250626 01:53:45.746345 5495 raft_consensus.cc:383] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:45.746387 5495 raft_consensus.cc:738] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19a270290fb44348ab21cff96a4f1e6, State: Initialized, Role: FOLLOWER
I20250626 01:53:45.746627 5495 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a270290fb44348ab21cff96a4f1e6" member_type: VOTER }
I20250626 01:53:45.746784 5495 raft_consensus.cc:397] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:53:45.746826 5495 raft_consensus.cc:491] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:53:45.746912 5495 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:45.748916 5495 raft_consensus.cc:513] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a270290fb44348ab21cff96a4f1e6" member_type: VOTER }
I20250626 01:53:45.749213 5495 leader_election.cc:304] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19a270290fb44348ab21cff96a4f1e6; no voters:
I20250626 01:53:45.749725 5495 leader_election.cc:290] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:53:45.749876 5498 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:45.750324 5498 raft_consensus.cc:695] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 1 LEADER]: Becoming Leader. State: Replica: a19a270290fb44348ab21cff96a4f1e6, State: Running, Role: LEADER
I20250626 01:53:45.750576 5498 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a270290fb44348ab21cff96a4f1e6" member_type: VOTER }
I20250626 01:53:45.750967 5495 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:53:45.751688 5499 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19a270290fb44348ab21cff96a4f1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a270290fb44348ab21cff96a4f1e6" member_type: VOTER } }
I20250626 01:53:45.751778 5500 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19a270290fb44348ab21cff96a4f1e6. Latest consensus state: current_term: 1 leader_uuid: "a19a270290fb44348ab21cff96a4f1e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19a270290fb44348ab21cff96a4f1e6" member_type: VOTER } }
I20250626 01:53:45.751976 5499 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [sys.catalog]: This master's current role is: LEADER
I20250626 01:53:45.752009 5500 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [sys.catalog]: This master's current role is: LEADER
I20250626 01:53:45.756384 5505 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:53:45.756553 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:53:45.757864 5505 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:53:45.761173 5505 catalog_manager.cc:1349] Generated new cluster ID: c8eb068b7e1d4a38adae029e36355a2f
I20250626 01:53:45.761286 5505 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:53:45.772738 5505 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:53:45.773566 5505 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:53:45.781658 5505 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6: Generated new TSK 0
I20250626 01:53:45.781939 5505 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:53:45.789587 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:45.792353 5516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.792573 5517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.794139 5519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:45.794174 413 server_base.cc:1048] running on GCE node
I20250626 01:53:45.794593 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:45.794660 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:45.794703 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902825794697 us; error 0 us; skew 500 ppm
I20250626 01:53:45.794870 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:45.796370 413 webserver.cc:469] Webserver started at http://127.0.103.65:35253/ using document root <none> and password file <none>
I20250626 01:53:45.796646 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:45.796729 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:45.796880 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:45.797721 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "d0bbbe8c617048749215332451248b02"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.800680 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:53:45.802567 5524 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.802930 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:53:45.803038 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "d0bbbe8c617048749215332451248b02"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.803154 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:45.813680 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:45.814306 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:45.814904 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:45.816119 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:45.816192 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.816267 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:45.816306 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.833060 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45837
I20250626 01:53:45.833084 5586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45837 every 8 connection(s)
I20250626 01:53:45.834803 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:45.838929 5592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.839116 5591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.839553 5594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:45.840001 413 server_base.cc:1048] running on GCE node
I20250626 01:53:45.840319 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:45.840397 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:45.840435 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902825840430 us; error 0 us; skew 500 ppm
I20250626 01:53:45.840624 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:45.842572 413 webserver.cc:469] Webserver started at http://127.0.103.66:33343/ using document root <none> and password file <none>
I20250626 01:53:45.842861 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:45.842969 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:45.843099 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:45.843449 5587 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36405
I20250626 01:53:45.843569 5587 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:45.843935 5587 heartbeater.cc:507] Master 127.0.103.126:36405 requested a full tablet report, sending...
I20250626 01:53:45.843961 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "3a8eaf231b8e40239e186c5ac550f1dd"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.844906 5460 ts_manager.cc:194] Registered new tserver with Master: d0bbbe8c617048749215332451248b02 (127.0.103.65:45837)
I20250626 01:53:45.845913 5460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43218
I20250626 01:53:45.846985 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250626 01:53:45.849051 5599 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.849586 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:53:45.849761 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "3a8eaf231b8e40239e186c5ac550f1dd"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.849886 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:45.862882 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:45.863516 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:45.864137 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:45.865036 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:45.865094 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.865170 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:45.865206 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.878413 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:37499
I20250626 01:53:45.878451 5661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:37499 every 8 connection(s)
I20250626 01:53:45.880015 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:45.882795 5666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.883283 5665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:45.883738 5668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:45.884917 413 server_base.cc:1048] running on GCE node
I20250626 01:53:45.885237 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:45.885306 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:45.885339 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902825885336 us; error 0 us; skew 500 ppm
I20250626 01:53:45.885505 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:45.886739 5662 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36405
I20250626 01:53:45.886860 5662 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:45.887037 413 webserver.cc:469] Webserver started at http://127.0.103.67:44223/ using document root <none> and password file <none>
I20250626 01:53:45.887248 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:45.887259 5662 heartbeater.cc:507] Master 127.0.103.126:36405 requested a full tablet report, sending...
I20250626 01:53:45.887322 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:45.887441 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:45.888175 5460 ts_manager.cc:194] Registered new tserver with Master: 3a8eaf231b8e40239e186c5ac550f1dd (127.0.103.66:37499)
I20250626 01:53:45.888183 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "3daacdc1b6a0462ca53760cb00a1edbc"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.888975 5460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43224
I20250626 01:53:45.891199 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250626 01:53:45.893260 5673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.893795 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:53:45.893896 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "3daacdc1b6a0462ca53760cb00a1edbc"
format_stamp: "Formatted at 2025-06-26 01:53:45 on dist-test-slave-psm5"
I20250626 01:53:45.893994 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:45.901711 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:45.902222 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:45.902756 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:45.903589 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:45.903645 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.903695 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:45.903738 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:45.916257 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:42925
I20250626 01:53:45.916296 5735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:42925 every 8 connection(s)
I20250626 01:53:45.922950 5736 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36405
I20250626 01:53:45.923102 5736 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:45.923435 5736 heartbeater.cc:507] Master 127.0.103.126:36405 requested a full tablet report, sending...
I20250626 01:53:45.924328 5460 ts_manager.cc:194] Registered new tserver with Master: 3daacdc1b6a0462ca53760cb00a1edbc (127.0.103.67:42925)
I20250626 01:53:45.925122 5460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43238
I20250626 01:53:45.925100 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007937354s
I20250626 01:53:46.847491 5587 heartbeater.cc:499] Master 127.0.103.126:36405 was elected leader, sending a full tablet report...
I20250626 01:53:46.890812 5662 heartbeater.cc:499] Master 127.0.103.126:36405 was elected leader, sending a full tablet report...
I20250626 01:53:46.926551 5736 heartbeater.cc:499] Master 127.0.103.126:36405 was elected leader, sending a full tablet report...
I20250626 01:53:46.954816 413 test_util.cc:276] Using random seed: -720801736
I20250626 01:53:46.968067 5460 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43244:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:53:46.968699 5460 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:53:46.998170 5701 tablet_service.cc:1468] Processing CreateTablet for tablet 88c05dd1108e413bbd4069e2ace1bdfc (DEFAULT_TABLE table=test-workload [id=554ed0ced92944b4accc06959a3aeec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:53:46.998170 5552 tablet_service.cc:1468] Processing CreateTablet for tablet 88c05dd1108e413bbd4069e2ace1bdfc (DEFAULT_TABLE table=test-workload [id=554ed0ced92944b4accc06959a3aeec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:53:46.998695 5701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88c05dd1108e413bbd4069e2ace1bdfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:46.998705 5552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88c05dd1108e413bbd4069e2ace1bdfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:47.004158 5757 tablet_bootstrap.cc:492] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc: Bootstrap starting.
I20250626 01:53:47.004307 5756 tablet_bootstrap.cc:492] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02: Bootstrap starting.
I20250626 01:53:47.006274 5756 tablet_bootstrap.cc:654] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:47.006176 5627 tablet_service.cc:1468] Processing CreateTablet for tablet 88c05dd1108e413bbd4069e2ace1bdfc (DEFAULT_TABLE table=test-workload [id=554ed0ced92944b4accc06959a3aeec8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250626 01:53:47.006455 5757 tablet_bootstrap.cc:654] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:47.006767 5627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88c05dd1108e413bbd4069e2ace1bdfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:47.008428 5756 tablet_bootstrap.cc:492] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02: No bootstrap required, opened a new log
I20250626 01:53:47.008450 5757 tablet_bootstrap.cc:492] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc: No bootstrap required, opened a new log
I20250626 01:53:47.008618 5757 ts_tablet_manager.cc:1397] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250626 01:53:47.008603 5756 ts_tablet_manager.cc:1397] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 01:53:47.009343 5757 raft_consensus.cc:357] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.009495 5757 raft_consensus.cc:383] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:47.009449 5756 raft_consensus.cc:357] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.009577 5757 raft_consensus.cc:738] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3daacdc1b6a0462ca53760cb00a1edbc, State: Initialized, Role: FOLLOWER
I20250626 01:53:47.009647 5756 raft_consensus.cc:383] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:47.009745 5756 raft_consensus.cc:738] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bbbe8c617048749215332451248b02, State: Initialized, Role: FOLLOWER
I20250626 01:53:47.009836 5757 consensus_queue.cc:260] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.009975 5756 consensus_queue.cc:260] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.016511 5757 ts_tablet_manager.cc:1428] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250626 01:53:47.020781 5756 ts_tablet_manager.cc:1428] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02: Time spent starting tablet: real 0.012s user 0.003s sys 0.007s
I20250626 01:53:47.021708 5762 tablet_bootstrap.cc:492] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd: Bootstrap starting.
I20250626 01:53:47.023653 5762 tablet_bootstrap.cc:654] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:47.026007 5762 tablet_bootstrap.cc:492] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd: No bootstrap required, opened a new log
I20250626 01:53:47.026193 5762 ts_tablet_manager.cc:1397] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250626 01:53:47.026932 5762 raft_consensus.cc:357] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.027120 5762 raft_consensus.cc:383] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:47.027212 5762 raft_consensus.cc:738] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a8eaf231b8e40239e186c5ac550f1dd, State: Initialized, Role: FOLLOWER
I20250626 01:53:47.027467 5762 consensus_queue.cc:260] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.028213 5762 ts_tablet_manager.cc:1428] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:53:47.032708 5764 raft_consensus.cc:491] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:47.032908 5764 raft_consensus.cc:513] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.033872 5764 leader_election.cc:290] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0bbbe8c617048749215332451248b02 (127.0.103.65:45837), 3daacdc1b6a0462ca53760cb00a1edbc (127.0.103.67:42925)
I20250626 01:53:47.039947 5711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88c05dd1108e413bbd4069e2ace1bdfc" candidate_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" is_pre_election: true
I20250626 01:53:47.040252 5711 raft_consensus.cc:2466] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8eaf231b8e40239e186c5ac550f1dd in term 0.
I20250626 01:53:47.040717 5602 leader_election.cc:304] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a8eaf231b8e40239e186c5ac550f1dd, 3daacdc1b6a0462ca53760cb00a1edbc; no voters:
I20250626 01:53:47.041006 5764 raft_consensus.cc:2802] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:47.041096 5764 raft_consensus.cc:491] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:47.041203 5764 raft_consensus.cc:3058] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:47.041294 5562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88c05dd1108e413bbd4069e2ace1bdfc" candidate_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bbbe8c617048749215332451248b02" is_pre_election: true
I20250626 01:53:47.041520 5562 raft_consensus.cc:2466] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a8eaf231b8e40239e186c5ac550f1dd in term 0.
I20250626 01:53:47.043327 5764 raft_consensus.cc:513] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.044271 5764 leader_election.cc:290] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [CANDIDATE]: Term 1 election: Requested vote from peers d0bbbe8c617048749215332451248b02 (127.0.103.65:45837), 3daacdc1b6a0462ca53760cb00a1edbc (127.0.103.67:42925)
I20250626 01:53:47.044345 5711 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88c05dd1108e413bbd4069e2ace1bdfc" candidate_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3daacdc1b6a0462ca53760cb00a1edbc"
I20250626 01:53:47.044577 5711 raft_consensus.cc:3058] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:47.044618 5562 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88c05dd1108e413bbd4069e2ace1bdfc" candidate_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0bbbe8c617048749215332451248b02"
I20250626 01:53:47.044795 5562 raft_consensus.cc:3058] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:47.046622 5711 raft_consensus.cc:2466] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8eaf231b8e40239e186c5ac550f1dd in term 1.
I20250626 01:53:47.046622 5562 raft_consensus.cc:2466] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a8eaf231b8e40239e186c5ac550f1dd in term 1.
I20250626 01:53:47.047122 5600 leader_election.cc:304] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a8eaf231b8e40239e186c5ac550f1dd, 3daacdc1b6a0462ca53760cb00a1edbc, d0bbbe8c617048749215332451248b02; no voters:
I20250626 01:53:47.047413 5764 raft_consensus.cc:2802] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:47.047830 5764 raft_consensus.cc:695] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 1 LEADER]: Becoming Leader. State: Replica: 3a8eaf231b8e40239e186c5ac550f1dd, State: Running, Role: LEADER
I20250626 01:53:47.048112 5764 consensus_queue.cc:237] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } }
I20250626 01:53:47.050609 5460 catalog_manager.cc:5582] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a8eaf231b8e40239e186c5ac550f1dd (127.0.103.66). New cstate: current_term: 1 leader_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0bbbe8c617048749215332451248b02" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 45837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a8eaf231b8e40239e186c5ac550f1dd" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37499 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:47.058012 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:53:47.065613 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:47.066020 413 tablet_replica.cc:331] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02: stopping tablet replica
I20250626 01:53:47.066211 413 raft_consensus.cc:2241] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:47.066402 413 raft_consensus.cc:2270] T 88c05dd1108e413bbd4069e2ace1bdfc P d0bbbe8c617048749215332451248b02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:47.080574 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:53:47.426606 5764 consensus_queue.cc:1035] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3daacdc1b6a0462ca53760cb00a1edbc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:53:47.434113 5600 consensus_peers.cc:487] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd -> Peer d0bbbe8c617048749215332451248b02 (127.0.103.65:45837): Couldn't send request to peer d0bbbe8c617048749215332451248b02. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:53:47.756501 5513 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:47.756966 5513 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:47.757067 5513 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:49.757411 5513 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:49.757879 5513 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:49.757985 5513 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 01:53:50.048986 5600 consensus_peers.cc:487] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd -> Peer d0bbbe8c617048749215332451248b02 (127.0.103.65:45837): Couldn't send request to peer d0bbbe8c617048749215332451248b02. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250626 01:53:51.037182 5600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20250626 01:53:51.758409 5513 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:51.758842 5513 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:51.758934 5513 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 01:53:52.600267 5600 consensus_peers.cc:487] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd -> Peer d0bbbe8c617048749215332451248b02 (127.0.103.65:45837): Couldn't send request to peer d0bbbe8c617048749215332451248b02. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250626 01:53:53.759439 5513 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:53.759963 5513 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:53.760056 5513 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 01:53:55.135481 5600 consensus_peers.cc:487] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd -> Peer d0bbbe8c617048749215332451248b02 (127.0.103.65:45837): Couldn't send request to peer d0bbbe8c617048749215332451248b02. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250626 01:53:55.760600 5513 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:55.761075 5513 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:55.761158 5513 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250626 01:53:56.109724 5600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250626 01:53:57.498418 5600 consensus_peers.cc:487] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd -> Peer d0bbbe8c617048749215332451248b02 (127.0.103.65:45837): Couldn't send request to peer d0bbbe8c617048749215332451248b02. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:45837: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250626 01:53:57.761590 5513 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250626 01:53:57.762081 5513 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250626 01:53:57.762171 5513 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250626 01:53:58.154170 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:53:58.178534 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:58.178836 413 tablet_replica.cc:331] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd: stopping tablet replica
I20250626 01:53:58.179030 413 raft_consensus.cc:2241] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:58.179363 413 raft_consensus.cc:2270] T 88c05dd1108e413bbd4069e2ace1bdfc P 3a8eaf231b8e40239e186c5ac550f1dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:58.196564 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:53:58.199501 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:53:58.207834 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:58.208184 413 tablet_replica.cc:331] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc: stopping tablet replica
I20250626 01:53:58.208369 413 raft_consensus.cc:2241] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:58.208552 413 raft_consensus.cc:2270] T 88c05dd1108e413bbd4069e2ace1bdfc P 3daacdc1b6a0462ca53760cb00a1edbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:58.222218 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:53:58.225603 413 master.cc:561] Master@127.0.103.126:36405 shutting down...
I20250626 01:53:58.234877 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:58.235196 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:58.235335 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P a19a270290fb44348ab21cff96a4f1e6: stopping tablet replica
I20250626 01:53:58.250029 413 master.cc:583] Master@127.0.103.126:36405 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12580 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250626 01:53:58.261413 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36757
I20250626 01:53:58.262002 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:58.264855 5797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.264880 5800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.264904 5798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:58.265175 413 server_base.cc:1048] running on GCE node
I20250626 01:53:58.266237 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:58.266326 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:58.266387 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902838266371 us; error 0 us; skew 500 ppm
I20250626 01:53:58.266530 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:58.272379 413 webserver.cc:469] Webserver started at http://127.0.103.126:38593/ using document root <none> and password file <none>
I20250626 01:53:58.272646 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:58.272727 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:58.272852 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:58.273521 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "9a69b06774a4419680f379f2b352b53a"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.276322 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250626 01:53:58.278194 5805 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.278580 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:53:58.278708 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "9a69b06774a4419680f379f2b352b53a"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.278841 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:58.295970 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:58.296500 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:58.308847 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36757
I20250626 01:53:58.308877 5856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36757 every 8 connection(s)
I20250626 01:53:58.310222 5857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:58.314517 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a: Bootstrap starting.
I20250626 01:53:58.316005 5857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:58.317719 5857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a: No bootstrap required, opened a new log
I20250626 01:53:58.318436 5857 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a69b06774a4419680f379f2b352b53a" member_type: VOTER }
I20250626 01:53:58.318581 5857 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:58.318643 5857 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a69b06774a4419680f379f2b352b53a, State: Initialized, Role: FOLLOWER
I20250626 01:53:58.318871 5857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a69b06774a4419680f379f2b352b53a" member_type: VOTER }
I20250626 01:53:58.319005 5857 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:53:58.319067 5857 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:53:58.319164 5857 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:58.321028 5857 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a69b06774a4419680f379f2b352b53a" member_type: VOTER }
I20250626 01:53:58.321240 5857 leader_election.cc:304] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a69b06774a4419680f379f2b352b53a; no voters:
I20250626 01:53:58.321703 5857 leader_election.cc:290] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:53:58.321854 5860 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:58.322367 5860 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 1 LEADER]: Becoming Leader. State: Replica: 9a69b06774a4419680f379f2b352b53a, State: Running, Role: LEADER
I20250626 01:53:58.322666 5857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:53:58.322655 5860 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a69b06774a4419680f379f2b352b53a" member_type: VOTER }
I20250626 01:53:58.323621 5861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a69b06774a4419680f379f2b352b53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a69b06774a4419680f379f2b352b53a" member_type: VOTER } }
I20250626 01:53:58.323693 5862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a69b06774a4419680f379f2b352b53a. Latest consensus state: current_term: 1 leader_uuid: "9a69b06774a4419680f379f2b352b53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a69b06774a4419680f379f2b352b53a" member_type: VOTER } }
I20250626 01:53:58.323935 5862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [sys.catalog]: This master's current role is: LEADER
I20250626 01:53:58.324285 5861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [sys.catalog]: This master's current role is: LEADER
I20250626 01:53:58.325331 5866 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:53:58.327471 5866 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:53:58.327848 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:53:58.330624 5866 catalog_manager.cc:1349] Generated new cluster ID: dc98eb83df644282a5e907e70f132770
I20250626 01:53:58.330737 5866 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:53:58.342039 5866 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:53:58.342859 5866 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:53:58.351298 5866 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a: Generated new TSK 0
I20250626 01:53:58.351565 5866 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:53:58.361212 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:58.364311 5878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.364775 5879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.364892 5881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:58.365284 413 server_base.cc:1048] running on GCE node
I20250626 01:53:58.365617 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:58.365698 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:58.365773 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902838365748 us; error 0 us; skew 500 ppm
I20250626 01:53:58.365958 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:58.367435 413 webserver.cc:469] Webserver started at http://127.0.103.65:45179/ using document root <none> and password file <none>
I20250626 01:53:58.367795 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:58.367940 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:58.368103 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:58.368795 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "6c2621e043b3442a99dc9711b8756b15"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.371563 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:53:58.373445 5886 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.373895 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:53:58.374027 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "6c2621e043b3442a99dc9711b8756b15"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.374155 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:58.394315 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:58.395008 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:58.395651 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:58.396642 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:58.396729 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.396833 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:58.396901 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.411217 5948 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41405 every 8 connection(s)
I20250626 01:53:58.411188 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41405
I20250626 01:53:58.413089 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:58.416150 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.416208 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.416815 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:58.417882 413 server_base.cc:1048] running on GCE node
I20250626 01:53:58.418191 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:58.418260 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:58.418323 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902838418299 us; error 0 us; skew 500 ppm
I20250626 01:53:58.418506 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:58.420274 413 webserver.cc:469] Webserver started at http://127.0.103.66:38073/ using document root <none> and password file <none>
I20250626 01:53:58.420540 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:58.420635 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:58.420722 5949 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36757
I20250626 01:53:58.420799 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:58.420881 5949 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:58.421267 5949 heartbeater.cc:507] Master 127.0.103.126:36757 requested a full tablet report, sending...
I20250626 01:53:58.421550 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "41392499c02a4004b367ed21fdac9455"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.422166 5822 ts_manager.cc:194] Registered new tserver with Master: 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405)
I20250626 01:53:58.423336 5822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58854
I20250626 01:53:58.424381 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:53:58.426111 5961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.426458 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20250626 01:53:58.426570 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "41392499c02a4004b367ed21fdac9455"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.426714 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:58.434582 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:58.435213 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:58.435806 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:58.436699 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:58.436779 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.436909 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:58.436971 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.452729 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:46391
I20250626 01:53:58.452730 6023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:46391 every 8 connection(s)
I20250626 01:53:58.454619 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:53:58.457759 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:53:58.458086 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:58.464807 413 server_base.cc:1048] running on GCE node
W20250626 01:53:58.465395 6031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:53:58.467377 6024 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36757
I20250626 01:53:58.467538 6024 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:58.467991 6024 heartbeater.cc:507] Master 127.0.103.126:36757 requested a full tablet report, sending...
I20250626 01:53:58.469254 5822 ts_manager.cc:194] Registered new tserver with Master: 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:53:58.470082 5822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58870
I20250626 01:53:58.477532 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:53:58.477653 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:53:58.477681 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902838477678 us; error 0 us; skew 500 ppm
I20250626 01:53:58.477841 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:53:58.479357 413 webserver.cc:469] Webserver started at http://127.0.103.67:37701/ using document root <none> and password file <none>
I20250626 01:53:58.479617 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:53:58.479693 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:53:58.479830 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:53:58.480499 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "727aec7109c14c1fbaad5982a2b87db9"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.483091 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20250626 01:53:58.484990 6035 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.485426 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:53:58.485527 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "727aec7109c14c1fbaad5982a2b87db9"
format_stamp: "Formatted at 2025-06-26 01:53:58 on dist-test-slave-psm5"
I20250626 01:53:58.485653 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:53:58.517828 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:53:58.518431 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:53:58.519045 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:53:58.519948 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:53:58.520010 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.520080 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:53:58.520123 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:53:58.533675 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:38585
I20250626 01:53:58.533704 6097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:38585 every 8 connection(s)
I20250626 01:53:58.540508 6098 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36757
I20250626 01:53:58.540652 6098 heartbeater.cc:461] Registering TS with master...
I20250626 01:53:58.540989 6098 heartbeater.cc:507] Master 127.0.103.126:36757 requested a full tablet report, sending...
I20250626 01:53:58.541846 5822 ts_manager.cc:194] Registered new tserver with Master: 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585)
I20250626 01:53:58.542620 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.008016029s
I20250626 01:53:58.542682 5822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58884
I20250626 01:53:59.424685 5949 heartbeater.cc:499] Master 127.0.103.126:36757 was elected leader, sending a full tablet report...
I20250626 01:53:59.471400 6024 heartbeater.cc:499] Master 127.0.103.126:36757 was elected leader, sending a full tablet report...
I20250626 01:53:59.543992 6098 heartbeater.cc:499] Master 127.0.103.126:36757 was elected leader, sending a full tablet report...
I20250626 01:53:59.571043 413 test_util.cc:276] Using random seed: -708185507
I20250626 01:53:59.584252 5822 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58892:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:53:59.585052 5822 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:53:59.619175 5989 tablet_service.cc:1468] Processing CreateTablet for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:53:59.619516 5987 tablet_service.cc:1468] Processing CreateTablet for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:53:59.619729 5989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.619954 5987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.622300 5986 tablet_service.cc:1468] Processing CreateTablet for tablet f08e6f44674145d59c3722bc731e0a85 (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:53:59.622783 5986 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08e6f44674145d59c3722bc731e0a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.619164 5988 tablet_service.cc:1468] Processing CreateTablet for tablet 8f182a244087461aad32fab0747c7e93 (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:53:59.623100 5988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f182a244087461aad32fab0747c7e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.626673 6118 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455: Bootstrap starting.
I20250626 01:53:59.627382 5914 tablet_service.cc:1468] Processing CreateTablet for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:53:59.627869 5914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.628986 6118 tablet_bootstrap.cc:654] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.630607 5913 tablet_service.cc:1468] Processing CreateTablet for tablet 8f182a244087461aad32fab0747c7e93 (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:53:59.631069 5913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f182a244087461aad32fab0747c7e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.631626 6063 tablet_service.cc:1468] Processing CreateTablet for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:53:59.632091 6063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.635372 6062 tablet_service.cc:1468] Processing CreateTablet for tablet 8f182a244087461aad32fab0747c7e93 (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:53:59.635988 6062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f182a244087461aad32fab0747c7e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.636307 5912 tablet_service.cc:1468] Processing CreateTablet for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:53:59.636735 5912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.640635 5911 tablet_service.cc:1468] Processing CreateTablet for tablet f08e6f44674145d59c3722bc731e0a85 (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:53:59.641415 5911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08e6f44674145d59c3722bc731e0a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.644129 6061 tablet_service.cc:1468] Processing CreateTablet for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:53:59.644716 6061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.651935 6060 tablet_service.cc:1468] Processing CreateTablet for tablet f08e6f44674145d59c3722bc731e0a85 (DEFAULT_TABLE table=test-workload [id=348f9f6f4a4a41a0a33c40ff48443b6f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:53:59.652457 6060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08e6f44674145d59c3722bc731e0a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:53:59.658324 6121 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: Bootstrap starting.
I20250626 01:53:59.660665 6121 tablet_bootstrap.cc:654] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.664178 6120 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9: Bootstrap starting.
I20250626 01:53:59.665825 6118 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455: No bootstrap required, opened a new log
I20250626 01:53:59.666013 6118 ts_tablet_manager.cc:1397] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.012s
I20250626 01:53:59.666958 6118 raft_consensus.cc:357] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.667174 6118 raft_consensus.cc:383] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.667237 6118 raft_consensus.cc:738] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.667550 6120 tablet_bootstrap.cc:654] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.667482 6118 consensus_queue.cc:260] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.668465 6118 ts_tablet_manager.cc:1428] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250626 01:53:59.668723 6121 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: No bootstrap required, opened a new log
I20250626 01:53:59.668795 6118 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: Bootstrap starting.
I20250626 01:53:59.668864 6121 ts_tablet_manager.cc:1397] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20250626 01:53:59.669821 6121 raft_consensus.cc:357] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.670023 6121 raft_consensus.cc:383] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.670099 6121 raft_consensus.cc:738] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2621e043b3442a99dc9711b8756b15, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.670368 6121 consensus_queue.cc:260] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.671214 6118 tablet_bootstrap.cc:654] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.676781 6120 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9: No bootstrap required, opened a new log
I20250626 01:53:59.676973 6120 ts_tablet_manager.cc:1397] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20250626 01:53:59.677809 6120 raft_consensus.cc:357] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.677978 6120 raft_consensus.cc:383] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.678045 6120 raft_consensus.cc:738] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727aec7109c14c1fbaad5982a2b87db9, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.678268 6120 consensus_queue.cc:260] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.679086 6120 ts_tablet_manager.cc:1428] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250626 01:53:59.679387 6120 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9: Bootstrap starting.
I20250626 01:53:59.679664 6118 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: No bootstrap required, opened a new log
I20250626 01:53:59.679849 6118 ts_tablet_manager.cc:1397] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20250626 01:53:59.681172 6118 raft_consensus.cc:357] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.681442 6118 raft_consensus.cc:383] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.681471 6120 tablet_bootstrap.cc:654] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.681542 6118 raft_consensus.cc:738] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.681912 6118 consensus_queue.cc:260] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.683095 6121 ts_tablet_manager.cc:1428] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: Time spent starting tablet: real 0.014s user 0.001s sys 0.013s
I20250626 01:53:59.683470 6118 ts_tablet_manager.cc:1428] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:53:59.683764 6118 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: Bootstrap starting.
I20250626 01:53:59.685258 6121 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15: Bootstrap starting.
I20250626 01:53:59.685740 6118 tablet_bootstrap.cc:654] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.687302 6121 tablet_bootstrap.cc:654] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.691625 6118 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: No bootstrap required, opened a new log
I20250626 01:53:59.691627 6121 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15: No bootstrap required, opened a new log
I20250626 01:53:59.691627 6120 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9: No bootstrap required, opened a new log
I20250626 01:53:59.691859 6118 ts_tablet_manager.cc:1397] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20250626 01:53:59.691859 6120 ts_tablet_manager.cc:1397] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20250626 01:53:59.691874 6121 ts_tablet_manager.cc:1397] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20250626 01:53:59.692891 6118 raft_consensus.cc:357] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.693007 6121 raft_consensus.cc:357] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.693126 6118 raft_consensus.cc:383] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.692891 6120 raft_consensus.cc:357] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.693181 6121 raft_consensus.cc:383] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.693332 6121 raft_consensus.cc:738] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2621e043b3442a99dc9711b8756b15, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.693179 6118 raft_consensus.cc:738] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.693354 6120 raft_consensus.cc:383] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.693552 6120 raft_consensus.cc:738] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727aec7109c14c1fbaad5982a2b87db9, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.693585 6121 consensus_queue.cc:260] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.693624 6118 consensus_queue.cc:260] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.693725 6120 consensus_queue.cc:260] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.694125 6121 ts_tablet_manager.cc:1428] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:53:59.694741 6121 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15: Bootstrap starting.
I20250626 01:53:59.694747 6120 ts_tablet_manager.cc:1428] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:53:59.694995 6120 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: Bootstrap starting.
I20250626 01:53:59.696794 6121 tablet_bootstrap.cc:654] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.696794 6120 tablet_bootstrap.cc:654] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.694125 6118 ts_tablet_manager.cc:1428] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:53:59.698226 6118 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: Bootstrap starting.
I20250626 01:53:59.700802 6118 tablet_bootstrap.cc:654] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.700834 6121 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15: No bootstrap required, opened a new log
I20250626 01:53:59.701011 6121 ts_tablet_manager.cc:1397] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20250626 01:53:59.701212 6120 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: No bootstrap required, opened a new log
I20250626 01:53:59.701360 6120 ts_tablet_manager.cc:1397] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250626 01:53:59.701769 6121 raft_consensus.cc:357] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.701951 6121 raft_consensus.cc:383] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.702018 6121 raft_consensus.cc:738] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2621e043b3442a99dc9711b8756b15, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.702175 6120 raft_consensus.cc:357] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.702332 6120 raft_consensus.cc:383] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.702263 6121 consensus_queue.cc:260] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.702402 6120 raft_consensus.cc:738] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727aec7109c14c1fbaad5982a2b87db9, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.702675 6120 consensus_queue.cc:260] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.703166 6120 ts_tablet_manager.cc:1428] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:53:59.703174 6118 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: No bootstrap required, opened a new log
I20250626 01:53:59.703368 6118 ts_tablet_manager.cc:1397] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250626 01:53:59.703469 6120 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9: Bootstrap starting.
I20250626 01:53:59.703831 6121 ts_tablet_manager.cc:1428] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250626 01:53:59.704155 6121 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15: Bootstrap starting.
I20250626 01:53:59.704757 6118 raft_consensus.cc:357] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.704952 6118 raft_consensus.cc:383] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.705034 6118 raft_consensus.cc:738] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.705300 6118 consensus_queue.cc:260] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.705458 6120 tablet_bootstrap.cc:654] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.705871 6118 ts_tablet_manager.cc:1428] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:53:59.706000 6121 tablet_bootstrap.cc:654] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15: Neither blocks nor log segments found. Creating new log.
I20250626 01:53:59.711498 6121 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15: No bootstrap required, opened a new log
I20250626 01:53:59.711635 6121 ts_tablet_manager.cc:1397] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20250626 01:53:59.712613 6121 raft_consensus.cc:357] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.712800 6121 raft_consensus.cc:383] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.712865 6121 raft_consensus.cc:738] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2621e043b3442a99dc9711b8756b15, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.713048 6121 consensus_queue.cc:260] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.713640 6121 ts_tablet_manager.cc:1428] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:53:59.714337 6120 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9: No bootstrap required, opened a new log
I20250626 01:53:59.714478 6120 ts_tablet_manager.cc:1397] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20250626 01:53:59.715281 6120 raft_consensus.cc:357] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.715425 6120 raft_consensus.cc:383] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:53:59.715487 6120 raft_consensus.cc:738] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 727aec7109c14c1fbaad5982a2b87db9, State: Initialized, Role: FOLLOWER
I20250626 01:53:59.715678 6120 consensus_queue.cc:260] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.716307 6120 ts_tablet_manager.cc:1428] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:53:59.772065 6123 raft_consensus.cc:491] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:59.772286 6123 raft_consensus.cc:513] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.773176 6123 leader_election.cc:290] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405), 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585)
I20250626 01:53:59.779572 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08e6f44674145d59c3722bc731e0a85" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9" is_pre_election: true
I20250626 01:53:59.779572 5924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08e6f44674145d59c3722bc731e0a85" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2621e043b3442a99dc9711b8756b15" is_pre_election: true
I20250626 01:53:59.780164 6073 raft_consensus.cc:2466] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 0.
I20250626 01:53:59.780202 5924 raft_consensus.cc:2466] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 0.
I20250626 01:53:59.780887 5963 leader_election.cc:304] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 6c2621e043b3442a99dc9711b8756b15, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.781231 6123 raft_consensus.cc:2802] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:59.781307 6123 raft_consensus.cc:491] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:59.781373 6123 raft_consensus.cc:3058] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.783397 6123 raft_consensus.cc:513] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.784155 6123 leader_election.cc:290] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405), 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585)
I20250626 01:53:59.784739 5924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08e6f44674145d59c3722bc731e0a85" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2621e043b3442a99dc9711b8756b15"
I20250626 01:53:59.784739 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f08e6f44674145d59c3722bc731e0a85" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9"
I20250626 01:53:59.784970 6073 raft_consensus.cc:3058] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.784978 5924 raft_consensus.cc:3058] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.787942 5924 raft_consensus.cc:2466] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 1.
I20250626 01:53:59.787942 6073 raft_consensus.cc:2466] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 1.
I20250626 01:53:59.788554 5963 leader_election.cc:304] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 6c2621e043b3442a99dc9711b8756b15, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.788944 6123 raft_consensus.cc:2802] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:59.789369 6123 raft_consensus.cc:695] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Becoming Leader. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Running, Role: LEADER
I20250626 01:53:59.789664 6123 consensus_queue.cc:237] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.792438 5822 catalog_manager.cc:5582] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41392499c02a4004b367ed21fdac9455 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:59.838317 6123 raft_consensus.cc:491] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:59.838533 6123 raft_consensus.cc:513] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.839166 6123 leader_election.cc:290] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405), 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585)
I20250626 01:53:59.839843 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f182a244087461aad32fab0747c7e93" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9" is_pre_election: true
I20250626 01:53:59.839843 5924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f182a244087461aad32fab0747c7e93" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2621e043b3442a99dc9711b8756b15" is_pre_election: true
I20250626 01:53:59.840142 5924 raft_consensus.cc:2466] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 0.
I20250626 01:53:59.840142 6073 raft_consensus.cc:2466] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 0.
I20250626 01:53:59.840633 5965 leader_election.cc:304] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.840905 6123 raft_consensus.cc:2802] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:59.840988 6123 raft_consensus.cc:491] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:59.841068 6123 raft_consensus.cc:3058] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.843076 6123 raft_consensus.cc:513] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.843739 6123 leader_election.cc:290] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405), 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585)
I20250626 01:53:59.844267 5924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f182a244087461aad32fab0747c7e93" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2621e043b3442a99dc9711b8756b15"
I20250626 01:53:59.844267 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8f182a244087461aad32fab0747c7e93" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9"
I20250626 01:53:59.844450 6073 raft_consensus.cc:3058] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.844448 5924 raft_consensus.cc:3058] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.846452 5924 raft_consensus.cc:2466] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 1.
I20250626 01:53:59.846467 6073 raft_consensus.cc:2466] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 1.
I20250626 01:53:59.846962 5963 leader_election.cc:304] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 6c2621e043b3442a99dc9711b8756b15, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.847244 6123 raft_consensus.cc:2802] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:59.847366 6123 raft_consensus.cc:695] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Becoming Leader. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Running, Role: LEADER
I20250626 01:53:59.847604 6123 consensus_queue.cc:237] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.850279 5822 catalog_manager.cc:5582] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41392499c02a4004b367ed21fdac9455 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: UNKNOWN } } }
I20250626 01:53:59.906366 6125 raft_consensus.cc:491] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:59.906591 6125 raft_consensus.cc:513] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.907440 6125 leader_election.cc:290] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585), 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:53:59.915167 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce6e8f5171f6498cbf2bdf7e4f2ad98c" candidate_uuid: "6c2621e043b3442a99dc9711b8756b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9" is_pre_election: true
I20250626 01:53:59.915431 6073 raft_consensus.cc:2466] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2621e043b3442a99dc9711b8756b15 in term 0.
I20250626 01:53:59.915701 5999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce6e8f5171f6498cbf2bdf7e4f2ad98c" candidate_uuid: "6c2621e043b3442a99dc9711b8756b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41392499c02a4004b367ed21fdac9455" is_pre_election: true
I20250626 01:53:59.915858 5890 leader_election.cc:304] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2621e043b3442a99dc9711b8756b15, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.916014 5999 raft_consensus.cc:2466] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2621e043b3442a99dc9711b8756b15 in term 0.
I20250626 01:53:59.916381 6125 raft_consensus.cc:2802] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:59.916471 6125 raft_consensus.cc:491] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:59.916780 6125 raft_consensus.cc:3058] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.920627 6125 raft_consensus.cc:513] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.921737 6125 leader_election.cc:290] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [CANDIDATE]: Term 1 election: Requested vote from peers 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585), 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:53:59.922446 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce6e8f5171f6498cbf2bdf7e4f2ad98c" candidate_uuid: "6c2621e043b3442a99dc9711b8756b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9"
I20250626 01:53:59.922688 6073 raft_consensus.cc:3058] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.923296 6127 raft_consensus.cc:491] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:53:59.923454 6127 raft_consensus.cc:513] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.923749 5999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce6e8f5171f6498cbf2bdf7e4f2ad98c" candidate_uuid: "6c2621e043b3442a99dc9711b8756b15" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41392499c02a4004b367ed21fdac9455"
I20250626 01:53:59.923944 5999 raft_consensus.cc:3058] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.924620 6127 leader_election.cc:290] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405), 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:53:59.927273 5999 raft_consensus.cc:2466] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2621e043b3442a99dc9711b8756b15 in term 1.
I20250626 01:53:59.928033 5887 leader_election.cc:304] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 6c2621e043b3442a99dc9711b8756b15; no voters:
I20250626 01:53:59.930044 6073 raft_consensus.cc:2466] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2621e043b3442a99dc9711b8756b15 in term 1.
I20250626 01:53:59.930966 6125 raft_consensus.cc:2802] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:59.932139 6125 raft_consensus.cc:695] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2621e043b3442a99dc9711b8756b15, State: Running, Role: LEADER
I20250626 01:53:59.932493 6125 consensus_queue.cc:237] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:53:59.933593 5924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de68e7691ff4390a0fcfcd59a0cf0fe" candidate_uuid: "727aec7109c14c1fbaad5982a2b87db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2621e043b3442a99dc9711b8756b15" is_pre_election: true
I20250626 01:53:59.934013 5924 raft_consensus.cc:2466] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 727aec7109c14c1fbaad5982a2b87db9 in term 0.
I20250626 01:53:59.934595 6037 leader_election.cc:304] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2621e043b3442a99dc9711b8756b15, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.934966 6127 raft_consensus.cc:2802] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:53:59.935089 6127 raft_consensus.cc:491] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:53:59.935218 6127 raft_consensus.cc:3058] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.936136 5822 catalog_manager.cc:5582] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "6c2621e043b3442a99dc9711b8756b15" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:59.937637 5999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de68e7691ff4390a0fcfcd59a0cf0fe" candidate_uuid: "727aec7109c14c1fbaad5982a2b87db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41392499c02a4004b367ed21fdac9455" is_pre_election: true
I20250626 01:53:59.937867 5999 raft_consensus.cc:2466] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 727aec7109c14c1fbaad5982a2b87db9 in term 0.
I20250626 01:53:59.938270 6127 raft_consensus.cc:513] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.938784 6127 leader_election.cc:290] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405), 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:53:59.939224 5999 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de68e7691ff4390a0fcfcd59a0cf0fe" candidate_uuid: "727aec7109c14c1fbaad5982a2b87db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41392499c02a4004b367ed21fdac9455"
I20250626 01:53:59.939240 5924 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4de68e7691ff4390a0fcfcd59a0cf0fe" candidate_uuid: "727aec7109c14c1fbaad5982a2b87db9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2621e043b3442a99dc9711b8756b15"
I20250626 01:53:59.939383 5999 raft_consensus.cc:3058] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.939460 5924 raft_consensus.cc:3058] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:53:59.942371 5999 raft_consensus.cc:2466] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 727aec7109c14c1fbaad5982a2b87db9 in term 1.
I20250626 01:53:59.942973 6036 leader_election.cc:304] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:53:59.942371 5924 raft_consensus.cc:2466] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 727aec7109c14c1fbaad5982a2b87db9 in term 1.
I20250626 01:53:59.943354 6127 raft_consensus.cc:2802] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:53:59.943833 6127 raft_consensus.cc:695] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 LEADER]: Becoming Leader. State: Replica: 727aec7109c14c1fbaad5982a2b87db9, State: Running, Role: LEADER
I20250626 01:53:59.944176 6127 consensus_queue.cc:237] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } }
I20250626 01:53:59.947424 5822 catalog_manager.cc:5582] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "727aec7109c14c1fbaad5982a2b87db9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } }
I20250626 01:53:59.978586 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:53:59.987229 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:53:59.987773 413 tablet_replica.cc:331] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15: stopping tablet replica
I20250626 01:53:59.988107 413 raft_consensus.cc:2241] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:59.988325 413 raft_consensus.cc:2270] T 8f182a244087461aad32fab0747c7e93 P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:59.989102 413 tablet_replica.cc:331] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15: stopping tablet replica
I20250626 01:53:59.989265 413 raft_consensus.cc:2241] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:59.989426 413 raft_consensus.cc:2270] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:59.989993 413 tablet_replica.cc:331] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: stopping tablet replica
I20250626 01:53:59.990144 413 raft_consensus.cc:2241] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:53:59.990375 413 pending_rounds.cc:70] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: Trying to abort 1 pending ops.
I20250626 01:53:59.990554 413 pending_rounds.cc:77] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171698032366804992 op_type: NO_OP noop_request { }
I20250626 01:53:59.990700 413 raft_consensus.cc:2887] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 01:53:59.990828 413 raft_consensus.cc:2270] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:53:59.991657 413 tablet_replica.cc:331] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15: stopping tablet replica
I20250626 01:53:59.991820 413 raft_consensus.cc:2241] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:53:59.991997 413 raft_consensus.cc:2270] T f08e6f44674145d59c3722bc731e0a85 P 6c2621e043b3442a99dc9711b8756b15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:00.007110 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:54:00.212205 6123 consensus_queue.cc:1035] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:00.218076 5963 consensus_peers.cc:487] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:00.272711 6123 consensus_queue.cc:1035] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:00.277987 5963 consensus_peers.cc:487] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:00.388525 6127 consensus_queue.cc:1035] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:00.394552 6037 consensus_peers.cc:487] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.012043 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:01.015077 6159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:01.015194 6160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:01.016131 413 server_base.cc:1048] running on GCE node
W20250626 01:54:01.016168 6162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:01.016568 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:01.016670 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:01.016744 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902841016723 us; error 0 us; skew 500 ppm
I20250626 01:54:01.016932 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:01.018534 413 webserver.cc:469] Webserver started at http://127.0.103.68:39203/ using document root <none> and password file <none>
I20250626 01:54:01.018788 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:01.018873 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:01.019026 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:01.019727 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "dd4c194209d24896a016b052ae21f4a4"
format_stamp: "Formatted at 2025-06-26 01:54:01 on dist-test-slave-psm5"
I20250626 01:54:01.022431 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:01.024191 6167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:01.024597 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:54:01.024726 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "dd4c194209d24896a016b052ae21f4a4"
format_stamp: "Formatted at 2025-06-26 01:54:01 on dist-test-slave-psm5"
I20250626 01:54:01.024871 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:01.041744 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:01.042388 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:01.043028 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:01.044008 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:01.044090 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:01.044196 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:01.044260 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:01.057915 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:46845
I20250626 01:54:01.057935 6229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:46845 every 8 connection(s)
I20250626 01:54:01.065150 6230 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36757
I20250626 01:54:01.065308 6230 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:01.065635 6230 heartbeater.cc:507] Master 127.0.103.126:36757 requested a full tablet report, sending...
I20250626 01:54:01.066474 5822 ts_manager.cc:194] Registered new tserver with Master: dd4c194209d24896a016b052ae21f4a4 (127.0.103.68:46845)
I20250626 01:54:01.067238 5822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58900
I20250626 01:54:01.138669 6154 consensus_queue.cc:579] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.291s)
W20250626 01:54:01.140110 5963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250626 01:54:01.143023 5999 consensus_queue.cc:237] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:01.144999 6073 raft_consensus.cc:1273] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:54:01.145587 6234 consensus_queue.cc:1035] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 01:54:01.146689 5963 consensus_peers.cc:487] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.148025 6235 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.148900 6073 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.152243 5806 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8f182a244087461aad32fab0747c7e93 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250626 01:54:01.152864 5962 consensus_peers.cc:487] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 -> Peer dd4c194209d24896a016b052ae21f4a4 (127.0.103.68:46845): Couldn't send request to peer dd4c194209d24896a016b052ae21f4a4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f182a244087461aad32fab0747c7e93. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.153916 5822 catalog_manager.cc:5582] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 reported cstate change: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:54:01.328866 5870 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6c2621e043b3442a99dc9711b8756b15 not available for placement
I20250626 01:54:01.354183 6235 consensus_queue.cc:579] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.564s)
I20250626 01:54:01.357748 5999 consensus_queue.cc:237] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:01.359819 6073 raft_consensus.cc:1273] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 01:54:01.360006 5962 consensus_peers.cc:487] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 -> Peer dd4c194209d24896a016b052ae21f4a4 (127.0.103.68:46845): Couldn't send request to peer dd4c194209d24896a016b052ae21f4a4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08e6f44674145d59c3722bc731e0a85. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.360384 6235 consensus_queue.cc:1035] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 01:54:01.360831 5963 consensus_peers.cc:487] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.362876 6154 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.363499 6073 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.365974 5806 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f08e6f44674145d59c3722bc731e0a85 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:54:01.367245 5822 catalog_manager.cc:5582] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 reported cstate change: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.542428 6147 consensus_queue.cc:579] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.598s)
I20250626 01:54:01.546908 6073 consensus_queue.cc:237] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
W20250626 01:54:01.550212 6037 consensus_peers.cc:487] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.550251 5999 raft_consensus.cc:1273] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Refusing update from remote peer 727aec7109c14c1fbaad5982a2b87db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:54:01.550798 6245 consensus_queue.cc:1035] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:54:01.553566 6147 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.556412 5809 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250626 01:54:01.557126 6036 consensus_peers.cc:487] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 -> Peer dd4c194209d24896a016b052ae21f4a4 (127.0.103.68:46845): Couldn't send request to peer dd4c194209d24896a016b052ae21f4a4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4de68e7691ff4390a0fcfcd59a0cf0fe. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:01.557066 5999 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:01.557796 5821 catalog_manager.cc:5582] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 reported cstate change: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "727aec7109c14c1fbaad5982a2b87db9" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:01.768306 6251 ts_tablet_manager.cc:927] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: Initiating tablet copy from peer 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:54:01.768865 6251 tablet_copy_client.cc:323] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:46391
I20250626 01:54:01.774974 6009 tablet_copy_service.cc:140] P 41392499c02a4004b367ed21fdac9455: Received BeginTabletCopySession request for tablet 8f182a244087461aad32fab0747c7e93 from peer dd4c194209d24896a016b052ae21f4a4 ({username='slave'} at 127.0.0.1:44954)
I20250626 01:54:01.775157 6009 tablet_copy_service.cc:161] P 41392499c02a4004b367ed21fdac9455: Beginning new tablet copy session on tablet 8f182a244087461aad32fab0747c7e93 from peer dd4c194209d24896a016b052ae21f4a4 at {username='slave'} at 127.0.0.1:44954: session id = dd4c194209d24896a016b052ae21f4a4-8f182a244087461aad32fab0747c7e93
I20250626 01:54:01.777236 6009 tablet_copy_source_session.cc:215] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:01.778424 6251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f182a244087461aad32fab0747c7e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:01.782984 6251 tablet_copy_client.cc:806] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:01.783236 6251 tablet_copy_client.cc:670] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:01.784801 6251 tablet_copy_client.cc:538] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:01.787266 6251 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: Bootstrap starting.
I20250626 01:54:01.791834 6251 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:01.792199 6251 tablet_bootstrap.cc:492] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: Bootstrap complete.
I20250626 01:54:01.792392 6251 ts_tablet_manager.cc:1397] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: Time spent bootstrapping tablet: real 0.005s user 0.007s sys 0.000s
I20250626 01:54:01.792963 6251 raft_consensus.cc:357] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:01.793113 6251 raft_consensus.cc:738] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd4c194209d24896a016b052ae21f4a4, State: Initialized, Role: LEARNER
I20250626 01:54:01.793246 6251 consensus_queue.cc:260] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:01.793975 6251 ts_tablet_manager.cc:1428] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:01.794054 6230 heartbeater.cc:499] Master 127.0.103.126:36757 was elected leader, sending a full tablet report...
I20250626 01:54:01.794771 6009 tablet_copy_service.cc:342] P 41392499c02a4004b367ed21fdac9455: Request end of tablet copy session dd4c194209d24896a016b052ae21f4a4-8f182a244087461aad32fab0747c7e93 received from {username='slave'} at 127.0.0.1:44954
I20250626 01:54:01.794899 6009 tablet_copy_service.cc:434] P 41392499c02a4004b367ed21fdac9455: ending tablet copy session dd4c194209d24896a016b052ae21f4a4-8f182a244087461aad32fab0747c7e93 on tablet 8f182a244087461aad32fab0747c7e93 with peer dd4c194209d24896a016b052ae21f4a4
I20250626 01:54:01.843914 6251 ts_tablet_manager.cc:927] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: Initiating tablet copy from peer 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:54:01.844522 6251 tablet_copy_client.cc:323] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:46391
I20250626 01:54:01.845263 6009 tablet_copy_service.cc:140] P 41392499c02a4004b367ed21fdac9455: Received BeginTabletCopySession request for tablet f08e6f44674145d59c3722bc731e0a85 from peer dd4c194209d24896a016b052ae21f4a4 ({username='slave'} at 127.0.0.1:44954)
I20250626 01:54:01.845413 6009 tablet_copy_service.cc:161] P 41392499c02a4004b367ed21fdac9455: Beginning new tablet copy session on tablet f08e6f44674145d59c3722bc731e0a85 from peer dd4c194209d24896a016b052ae21f4a4 at {username='slave'} at 127.0.0.1:44954: session id = dd4c194209d24896a016b052ae21f4a4-f08e6f44674145d59c3722bc731e0a85
I20250626 01:54:01.847412 6009 tablet_copy_source_session.cc:215] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:01.848623 6251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08e6f44674145d59c3722bc731e0a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:01.853063 6251 tablet_copy_client.cc:806] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:01.853319 6251 tablet_copy_client.cc:670] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:01.854840 6251 tablet_copy_client.cc:538] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:01.857599 6251 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: Bootstrap starting.
I20250626 01:54:01.862491 6251 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:01.862809 6251 tablet_bootstrap.cc:492] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: Bootstrap complete.
I20250626 01:54:01.863035 6251 ts_tablet_manager.cc:1397] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250626 01:54:01.863620 6251 raft_consensus.cc:357] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:01.863776 6251 raft_consensus.cc:738] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd4c194209d24896a016b052ae21f4a4, State: Initialized, Role: LEARNER
I20250626 01:54:01.863974 6251 consensus_queue.cc:260] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:01.864398 6251 ts_tablet_manager.cc:1428] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20250626 01:54:01.865223 6009 tablet_copy_service.cc:342] P 41392499c02a4004b367ed21fdac9455: Request end of tablet copy session dd4c194209d24896a016b052ae21f4a4-f08e6f44674145d59c3722bc731e0a85 received from {username='slave'} at 127.0.0.1:44954
I20250626 01:54:01.865319 6009 tablet_copy_service.cc:434] P 41392499c02a4004b367ed21fdac9455: ending tablet copy session dd4c194209d24896a016b052ae21f4a4-f08e6f44674145d59c3722bc731e0a85 on tablet f08e6f44674145d59c3722bc731e0a85 with peer dd4c194209d24896a016b052ae21f4a4
I20250626 01:54:01.873351 6234 raft_consensus.cc:491] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:01.873544 6234 raft_consensus.cc:513] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:54:01.874205 6234 leader_election.cc:290] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585), 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405)
I20250626 01:54:01.874733 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce6e8f5171f6498cbf2bdf7e4f2ad98c" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9" is_pre_election: true
I20250626 01:54:01.874956 6073 raft_consensus.cc:2466] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 1.
I20250626 01:54:01.875345 5965 leader_election.cc:304] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 727aec7109c14c1fbaad5982a2b87db9; no voters:
I20250626 01:54:01.875566 6234 raft_consensus.cc:2802] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:54:01.875782 6234 raft_consensus.cc:491] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250626 01:54:01.875749 5963 leader_election.cc:336] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
I20250626 01:54:01.875906 6234 raft_consensus.cc:3058] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:01.877911 6234 raft_consensus.cc:513] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:54:01.878540 6234 leader_election.cc:290] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 2 election: Requested vote from peers 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585), 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405)
I20250626 01:54:01.879001 6073 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ce6e8f5171f6498cbf2bdf7e4f2ad98c" candidate_uuid: "41392499c02a4004b367ed21fdac9455" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "727aec7109c14c1fbaad5982a2b87db9"
I20250626 01:54:01.879179 6073 raft_consensus.cc:3058] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Advancing to term 2
W20250626 01:54:01.879992 5963 leader_election.cc:336] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
I20250626 01:54:01.881425 6073 raft_consensus.cc:2466] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41392499c02a4004b367ed21fdac9455 in term 2.
I20250626 01:54:01.881814 5965 leader_election.cc:304] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41392499c02a4004b367ed21fdac9455, 727aec7109c14c1fbaad5982a2b87db9; no voters: 6c2621e043b3442a99dc9711b8756b15
I20250626 01:54:01.882092 6234 raft_consensus.cc:2802] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:54:01.882418 6234 raft_consensus.cc:695] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 LEADER]: Becoming Leader. State: Replica: 41392499c02a4004b367ed21fdac9455, State: Running, Role: LEADER
I20250626 01:54:01.882701 6234 consensus_queue.cc:237] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } }
I20250626 01:54:01.885200 5821 catalog_manager.cc:5582] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 reported cstate change: term changed from 1 to 2, leader changed from 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) to 41392499c02a4004b367ed21fdac9455 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:02.150969 6234 consensus_queue.cc:579] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.008s)
I20250626 01:54:02.167980 6251 ts_tablet_manager.cc:927] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: Initiating tablet copy from peer 727aec7109c14c1fbaad5982a2b87db9 (127.0.103.67:38585)
I20250626 01:54:02.168596 6251 tablet_copy_client.cc:323] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:38585
I20250626 01:54:02.174180 6083 tablet_copy_service.cc:140] P 727aec7109c14c1fbaad5982a2b87db9: Received BeginTabletCopySession request for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe from peer dd4c194209d24896a016b052ae21f4a4 ({username='slave'} at 127.0.0.1:46004)
I20250626 01:54:02.174355 6083 tablet_copy_service.cc:161] P 727aec7109c14c1fbaad5982a2b87db9: Beginning new tablet copy session on tablet 4de68e7691ff4390a0fcfcd59a0cf0fe from peer dd4c194209d24896a016b052ae21f4a4 at {username='slave'} at 127.0.0.1:46004: session id = dd4c194209d24896a016b052ae21f4a4-4de68e7691ff4390a0fcfcd59a0cf0fe
I20250626 01:54:02.176285 6083 tablet_copy_source_session.cc:215] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:02.177587 6251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:02.182090 6251 tablet_copy_client.cc:806] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:02.182349 6251 tablet_copy_client.cc:670] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:02.184082 6251 tablet_copy_client.cc:538] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:02.186623 6251 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: Bootstrap starting.
I20250626 01:54:02.191437 6251 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:02.191785 6251 tablet_bootstrap.cc:492] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: Bootstrap complete.
I20250626 01:54:02.192077 6251 ts_tablet_manager.cc:1397] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.004s
I20250626 01:54:02.192735 6251 raft_consensus.cc:357] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:02.192893 6251 raft_consensus.cc:738] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dd4c194209d24896a016b052ae21f4a4, State: Initialized, Role: LEARNER
I20250626 01:54:02.193077 6251 consensus_queue.cc:260] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:02.193514 6251 ts_tablet_manager.cc:1428] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:54:02.194281 6083 tablet_copy_service.cc:342] P 727aec7109c14c1fbaad5982a2b87db9: Request end of tablet copy session dd4c194209d24896a016b052ae21f4a4-4de68e7691ff4390a0fcfcd59a0cf0fe received from {username='slave'} at 127.0.0.1:46004
I20250626 01:54:02.194382 6083 tablet_copy_service.cc:434] P 727aec7109c14c1fbaad5982a2b87db9: ending tablet copy session dd4c194209d24896a016b052ae21f4a4-4de68e7691ff4390a0fcfcd59a0cf0fe on tablet 4de68e7691ff4390a0fcfcd59a0cf0fe with peer dd4c194209d24896a016b052ae21f4a4
I20250626 01:54:02.223300 6205 raft_consensus.cc:1215] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 01:54:02.280097 6073 raft_consensus.cc:1273] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 01:54:02.280651 6259 consensus_queue.cc:1035] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:02.285873 5963 consensus_peers.cc:487] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:02.297421 6205 raft_consensus.cc:1215] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250626 01:54:02.329743 5870 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6c2621e043b3442a99dc9711b8756b15 not available for placement
I20250626 01:54:02.391173 6234 consensus_queue.cc:579] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.033s)
I20250626 01:54:02.613755 6261 consensus_queue.cc:579] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.066s)
I20250626 01:54:02.661654 6264 raft_consensus.cc:1062] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: attempting to promote NON_VOTER dd4c194209d24896a016b052ae21f4a4 to VOTER
I20250626 01:54:02.662382 6264 consensus_queue.cc:237] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:02.664793 6205 raft_consensus.cc:1273] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:02.664866 6073 raft_consensus.cc:1273] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:02.665334 6264 consensus_queue.cc:1035] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 01:54:02.665794 5963 consensus_peers.cc:487] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:02.665810 6276 consensus_queue.cc:1035] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:02.668231 6264 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.668921 6205 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.672590 6073 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.673961 5822 catalog_manager.cc:5582] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 reported cstate change: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:54:02.679466 5999 consensus_queue.cc:237] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:02.681373 6073 raft_consensus.cc:1273] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:02.681373 6205 raft_consensus.cc:1273] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:02.681955 6276 consensus_queue.cc:1035] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:02.682160 6234 consensus_queue.cc:1035] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:02.684234 6234 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.685014 6073 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.685141 6205 raft_consensus.cc:2953] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.687167 5806 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 8f182a244087461aad32fab0747c7e93 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:02.688557 5821 catalog_manager.cc:5582] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 reported cstate change: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 01:54:02.694301 5807 catalog_manager.cc:4671] TS 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f182a244087461aad32fab0747c7e93: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
I20250626 01:54:02.696722 6205 raft_consensus.cc:1215] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250626 01:54:02.723363 6234 raft_consensus.cc:1062] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: attempting to promote NON_VOTER dd4c194209d24896a016b052ae21f4a4 to VOTER
I20250626 01:54:02.724125 6234 consensus_queue.cc:237] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:02.726341 6073 raft_consensus.cc:1273] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:02.726536 6205 raft_consensus.cc:1273] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:02.726890 6234 consensus_queue.cc:1035] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:02.727057 6259 consensus_queue.cc:1035] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 01:54:02.727296 5963 consensus_peers.cc:487] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:02.730240 6259 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.730916 6205 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.735286 6073 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.736013 5821 catalog_manager.cc:5582] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 reported cstate change: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.741607 5999 consensus_queue.cc:237] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:02.743256 6073 raft_consensus.cc:1273] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:02.743363 6205 raft_consensus.cc:1273] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:02.743837 6264 consensus_queue.cc:1035] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:02.744103 6259 consensus_queue.cc:1035] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:02.746210 6234 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.746874 6073 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.749409 5806 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f08e6f44674145d59c3722bc731e0a85 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:02.750037 6205 raft_consensus.cc:2953] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:02.750874 5821 catalog_manager.cc:5582] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 reported cstate change: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:54:03.172807 6271 raft_consensus.cc:1062] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: attempting to promote NON_VOTER dd4c194209d24896a016b052ae21f4a4 to VOTER
I20250626 01:54:03.173548 6271 consensus_queue.cc:237] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:03.175974 6205 raft_consensus.cc:1273] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 LEARNER]: Refusing update from remote peer 727aec7109c14c1fbaad5982a2b87db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:03.176177 5999 raft_consensus.cc:1273] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Refusing update from remote peer 727aec7109c14c1fbaad5982a2b87db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:03.176564 6288 consensus_queue.cc:1035] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:03.176844 6271 consensus_queue.cc:1035] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 01:54:03.177093 6037 consensus_peers.cc:487] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:03.179689 6288 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:03.180344 5999 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:03.184170 6205 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:03.184397 5821 catalog_manager.cc:5582] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 reported cstate change: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "727aec7109c14c1fbaad5982a2b87db9" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:54:03.190497 6073 consensus_queue.cc:237] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:03.192111 5999 raft_consensus.cc:1273] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Refusing update from remote peer 727aec7109c14c1fbaad5982a2b87db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:03.192111 6205 raft_consensus.cc:1273] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Refusing update from remote peer 727aec7109c14c1fbaad5982a2b87db9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:03.192584 6261 consensus_queue.cc:1035] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:03.192768 6272 consensus_queue.cc:1035] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:03.193346 6259 consensus_queue.cc:579] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.310s)
I20250626 01:54:03.195521 6271 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:03.198401 5999 consensus_queue.cc:237] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:03.199383 5809 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:03.199504 5998 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:03.199692 6205 raft_consensus.cc:2953] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:03.200316 5822 catalog_manager.cc:5582] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 reported cstate change: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "727aec7109c14c1fbaad5982a2b87db9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 01:54:03.203871 5962 consensus_peers.cc:487] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 -> Peer dd4c194209d24896a016b052ae21f4a4 (127.0.103.68:46845): Couldn't send request to peer dd4c194209d24896a016b052ae21f4a4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce6e8f5171f6498cbf2bdf7e4f2ad98c. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:03.205488 6073 raft_consensus.cc:1273] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250626 01:54:03.206408 5963 consensus_peers.cc:487] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:03.206580 6234 consensus_queue.cc:1035] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:54:03.209236 6259 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:03.209939 6073 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } } }
I20250626 01:54:03.212675 5806 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:54:03.213881 5821 catalog_manager.cc:5582] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 reported cstate change: config changed from index -1 to 2, NON_VOTER dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:54:03.330593 5870 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6c2621e043b3442a99dc9711b8756b15 not available for placement
I20250626 01:54:03.664121 6291 ts_tablet_manager.cc:927] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: Initiating tablet copy from peer 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391)
I20250626 01:54:03.664865 6291 tablet_copy_client.cc:323] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:46391
I20250626 01:54:03.665521 6009 tablet_copy_service.cc:140] P 41392499c02a4004b367ed21fdac9455: Received BeginTabletCopySession request for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c from peer dd4c194209d24896a016b052ae21f4a4 ({username='slave'} at 127.0.0.1:44954)
I20250626 01:54:03.665670 6009 tablet_copy_service.cc:161] P 41392499c02a4004b367ed21fdac9455: Beginning new tablet copy session on tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c from peer dd4c194209d24896a016b052ae21f4a4 at {username='slave'} at 127.0.0.1:44954: session id = dd4c194209d24896a016b052ae21f4a4-ce6e8f5171f6498cbf2bdf7e4f2ad98c
I20250626 01:54:03.668258 6009 tablet_copy_source_session.cc:215] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:03.669523 6291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:03.674198 6291 tablet_copy_client.cc:806] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:03.674538 6291 tablet_copy_client.cc:670] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:03.676091 6291 tablet_copy_client.cc:538] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:03.678704 6291 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: Bootstrap starting.
I20250626 01:54:03.683553 6291 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:03.684015 6291 tablet_bootstrap.cc:492] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: Bootstrap complete.
I20250626 01:54:03.684265 6291 ts_tablet_manager.cc:1397] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.003s
I20250626 01:54:03.684921 6291 raft_consensus.cc:357] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:03.685084 6291 raft_consensus.cc:738] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dd4c194209d24896a016b052ae21f4a4, State: Initialized, Role: LEARNER
I20250626 01:54:03.685281 6291 consensus_queue.cc:260] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: true } }
I20250626 01:54:03.685746 6291 ts_tablet_manager.cc:1428] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:54:03.686646 6009 tablet_copy_service.cc:342] P 41392499c02a4004b367ed21fdac9455: Request end of tablet copy session dd4c194209d24896a016b052ae21f4a4-ce6e8f5171f6498cbf2bdf7e4f2ad98c received from {username='slave'} at 127.0.0.1:44954
I20250626 01:54:03.686779 6009 tablet_copy_service.cc:434] P 41392499c02a4004b367ed21fdac9455: ending tablet copy session dd4c194209d24896a016b052ae21f4a4-ce6e8f5171f6498cbf2bdf7e4f2ad98c on tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c with peer dd4c194209d24896a016b052ae21f4a4
W20250626 01:54:03.834566 5807 catalog_manager.cc:4671] TS 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 4de68e7691ff4390a0fcfcd59a0cf0fe: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
I20250626 01:54:04.113782 6205 raft_consensus.cc:1215] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250626 01:54:04.266923 6264 consensus_queue.cc:579] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Leader has been unable to successfully communicate with peer 6c2621e043b3442a99dc9711b8756b15 for more than 1 seconds (1.068s)
W20250626 01:54:04.331487 5870 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6c2621e043b3442a99dc9711b8756b15 not available for placement
I20250626 01:54:04.595717 6264 raft_consensus.cc:1062] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: attempting to promote NON_VOTER dd4c194209d24896a016b052ae21f4a4 to VOTER
I20250626 01:54:04.596459 6264 consensus_queue.cc:237] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:04.598783 6205 raft_consensus.cc:1273] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 LEARNER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:54:04.599133 6073 raft_consensus.cc:1273] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:54:04.599328 6299 consensus_queue.cc:1035] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:04.599634 6264 consensus_queue.cc:1035] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250626 01:54:04.600952 5963 consensus_peers.cc:487] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 -> Peer 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): Couldn't send request to peer 6c2621e043b3442a99dc9711b8756b15. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:04.602149 6264 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.602766 6205 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.607136 6073 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.607722 5821 catalog_manager.cc:5582] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 reported cstate change: config changed from index 2 to 3, dd4c194209d24896a016b052ae21f4a4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "6c2621e043b3442a99dc9711b8756b15" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41405 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.614251 5999 consensus_queue.cc:237] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } }
I20250626 01:54:04.615996 6205 raft_consensus.cc:1273] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:54:04.616125 6073 raft_consensus.cc:1273] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Refusing update from remote peer 41392499c02a4004b367ed21fdac9455: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:54:04.616549 6297 consensus_queue.cc:1035] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:04.616825 6264 consensus_queue.cc:1035] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:04.618993 6264 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.619586 6205 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.619715 6073 raft_consensus.cc:2953] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } } }
I20250626 01:54:04.622226 5806 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:04.624884 5822 catalog_manager.cc:5582] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 reported cstate change: config changed from index 3 to 4, VOTER 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "41392499c02a4004b367ed21fdac9455" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "727aec7109c14c1fbaad5982a2b87db9" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "41392499c02a4004b367ed21fdac9455" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd4c194209d24896a016b052ae21f4a4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 46845 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250626 01:54:04.969928 5807 catalog_manager.cc:4671] TS 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f182a244087461aad32fab0747c7e93: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
W20250626 01:54:06.810464 5807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20250626 01:54:06.811729 5807 catalog_manager.cc:4671] TS 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
I20250626 01:54:08.086220 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:54:08.094396 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:08.094730 413 tablet_replica.cc:331] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455: stopping tablet replica
I20250626 01:54:08.094939 413 raft_consensus.cc:2241] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:08.095297 413 raft_consensus.cc:2270] T 8f182a244087461aad32fab0747c7e93 P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.096016 413 tablet_replica.cc:331] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455: stopping tablet replica
I20250626 01:54:08.096186 413 raft_consensus.cc:2241] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:54:08.096457 413 raft_consensus.cc:2270] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 41392499c02a4004b367ed21fdac9455 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.096957 413 tablet_replica.cc:331] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455: stopping tablet replica
I20250626 01:54:08.097103 413 raft_consensus.cc:2241] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.097327 413 raft_consensus.cc:2270] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.097779 413 tablet_replica.cc:331] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455: stopping tablet replica
I20250626 01:54:08.097919 413 raft_consensus.cc:2241] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:08.098169 413 raft_consensus.cc:2270] T f08e6f44674145d59c3722bc731e0a85 P 41392499c02a4004b367ed21fdac9455 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250626 01:54:08.111871 6036 consensus_peers.cc:487] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 -> Peer 41392499c02a4004b367ed21fdac9455 (127.0.103.66:46391): Couldn't send request to peer 41392499c02a4004b367ed21fdac9455. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.66:46391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:08.112505 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:54:08.127381 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:54:08.136507 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:08.136745 413 tablet_replica.cc:331] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9: stopping tablet replica
I20250626 01:54:08.136957 413 raft_consensus.cc:2241] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.137218 413 raft_consensus.cc:2270] T f08e6f44674145d59c3722bc731e0a85 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.137804 413 tablet_replica.cc:331] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9: stopping tablet replica
I20250626 01:54:08.137939 413 raft_consensus.cc:2241] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:08.138217 413 raft_consensus.cc:2270] T 4de68e7691ff4390a0fcfcd59a0cf0fe P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.138666 413 tablet_replica.cc:331] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9: stopping tablet replica
I20250626 01:54:08.138788 413 raft_consensus.cc:2241] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.138999 413 raft_consensus.cc:2270] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P 727aec7109c14c1fbaad5982a2b87db9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.139415 413 tablet_replica.cc:331] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9: stopping tablet replica
I20250626 01:54:08.139516 413 raft_consensus.cc:2241] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.139726 413 raft_consensus.cc:2270] T 8f182a244087461aad32fab0747c7e93 P 727aec7109c14c1fbaad5982a2b87db9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.154371 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:54:08.158023 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:54:08.167009 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:08.167265 413 tablet_replica.cc:331] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4: stopping tablet replica
I20250626 01:54:08.167452 413 raft_consensus.cc:2241] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.167670 413 raft_consensus.cc:2270] T ce6e8f5171f6498cbf2bdf7e4f2ad98c P dd4c194209d24896a016b052ae21f4a4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.168284 413 tablet_replica.cc:331] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4: stopping tablet replica
I20250626 01:54:08.168397 413 raft_consensus.cc:2241] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.168552 413 raft_consensus.cc:2270] T 4de68e7691ff4390a0fcfcd59a0cf0fe P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.168946 413 tablet_replica.cc:331] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4: stopping tablet replica
I20250626 01:54:08.169045 413 raft_consensus.cc:2241] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.169185 413 raft_consensus.cc:2270] T 8f182a244087461aad32fab0747c7e93 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.169538 413 tablet_replica.cc:331] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4: stopping tablet replica
I20250626 01:54:08.169629 413 raft_consensus.cc:2241] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:08.169778 413 raft_consensus.cc:2270] T f08e6f44674145d59c3722bc731e0a85 P dd4c194209d24896a016b052ae21f4a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:08.184815 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:54:08.188616 413 master.cc:561] Master@127.0.103.126:36757 shutting down...
W20250626 01:54:08.909077 5807 catalog_manager.cc:4671] TS 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ce6e8f5171f6498cbf2bdf7e4f2ad98c: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
W20250626 01:54:11.161322 5807 catalog_manager.cc:4671] TS 6c2621e043b3442a99dc9711b8756b15 (127.0.103.65:41405): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8f182a244087461aad32fab0747c7e93: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41405: connect: Connection refused (error 111)
I20250626 01:54:11.789502 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:11.789846 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:11.789978 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9a69b06774a4419680f379f2b352b53a: stopping tablet replica
I20250626 01:54:11.815806 413 master.cc:583] Master@127.0.103.126:36757 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13570 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250626 01:54:11.831359 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:40787
I20250626 01:54:11.831964 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:11.834650 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:11.834650 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:11.835433 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:11.836077 413 server_base.cc:1048] running on GCE node
I20250626 01:54:11.836326 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:11.836401 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:11.836463 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902851836443 us; error 0 us; skew 500 ppm
I20250626 01:54:11.836651 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:11.837981 413 webserver.cc:469] Webserver started at http://127.0.103.126:37035/ using document root <none> and password file <none>
I20250626 01:54:11.838214 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:11.838294 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:11.838438 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:11.839123 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "348701fbef564bc3b08a7b4722cb0f5a"
format_stamp: "Formatted at 2025-06-26 01:54:11 on dist-test-slave-psm5"
I20250626 01:54:11.841920 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250626 01:54:11.843705 6323 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:11.844146 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:54:11.844259 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "348701fbef564bc3b08a7b4722cb0f5a"
format_stamp: "Formatted at 2025-06-26 01:54:11 on dist-test-slave-psm5"
I20250626 01:54:11.844384 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:11.868148 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:11.868741 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:11.879967 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:40787
I20250626 01:54:11.880000 6374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:40787 every 8 connection(s)
I20250626 01:54:11.881342 6375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:11.885679 6375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a: Bootstrap starting.
I20250626 01:54:11.886950 6375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:11.888612 6375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a: No bootstrap required, opened a new log
I20250626 01:54:11.889268 6375 raft_consensus.cc:357] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348701fbef564bc3b08a7b4722cb0f5a" member_type: VOTER }
I20250626 01:54:11.889410 6375 raft_consensus.cc:383] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:11.889467 6375 raft_consensus.cc:738] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348701fbef564bc3b08a7b4722cb0f5a, State: Initialized, Role: FOLLOWER
I20250626 01:54:11.889678 6375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348701fbef564bc3b08a7b4722cb0f5a" member_type: VOTER }
I20250626 01:54:11.889812 6375 raft_consensus.cc:397] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:54:11.889875 6375 raft_consensus.cc:491] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:54:11.889957 6375 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:11.891784 6375 raft_consensus.cc:513] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348701fbef564bc3b08a7b4722cb0f5a" member_type: VOTER }
I20250626 01:54:11.892040 6375 leader_election.cc:304] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 348701fbef564bc3b08a7b4722cb0f5a; no voters:
I20250626 01:54:11.892454 6375 leader_election.cc:290] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:54:11.892601 6378 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:11.893062 6378 raft_consensus.cc:695] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 1 LEADER]: Becoming Leader. State: Replica: 348701fbef564bc3b08a7b4722cb0f5a, State: Running, Role: LEADER
I20250626 01:54:11.893302 6378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348701fbef564bc3b08a7b4722cb0f5a" member_type: VOTER }
I20250626 01:54:11.893355 6375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:54:11.894203 6379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "348701fbef564bc3b08a7b4722cb0f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348701fbef564bc3b08a7b4722cb0f5a" member_type: VOTER } }
I20250626 01:54:11.894272 6380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 348701fbef564bc3b08a7b4722cb0f5a. Latest consensus state: current_term: 1 leader_uuid: "348701fbef564bc3b08a7b4722cb0f5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "348701fbef564bc3b08a7b4722cb0f5a" member_type: VOTER } }
I20250626 01:54:11.894564 6380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [sys.catalog]: This master's current role is: LEADER
I20250626 01:54:11.894644 6379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [sys.catalog]: This master's current role is: LEADER
I20250626 01:54:11.895854 6384 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:54:11.898088 6384 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:54:11.898721 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:54:11.901234 6384 catalog_manager.cc:1349] Generated new cluster ID: de5251dbf458420892ea144cbb9f933f
I20250626 01:54:11.901345 6384 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:54:11.918251 6384 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:54:11.919114 6384 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:54:11.930383 6384 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a: Generated new TSK 0
I20250626 01:54:11.930680 6384 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:54:11.964305 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:11.967532 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:11.967778 6397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:11.968207 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:11.987078 413 server_base.cc:1048] running on GCE node
I20250626 01:54:11.987425 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:11.987505 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:11.987603 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902851987552 us; error 0 us; skew 500 ppm
I20250626 01:54:11.987797 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:11.989298 413 webserver.cc:469] Webserver started at http://127.0.103.65:36213/ using document root <none> and password file <none>
I20250626 01:54:11.989552 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:11.989645 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:11.989791 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:11.990478 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "a7ce92ff0dd04c5fb105d084915365ef"
format_stamp: "Formatted at 2025-06-26 01:54:11 on dist-test-slave-psm5"
I20250626 01:54:11.993187 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20250626 01:54:12.009181 6404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.009569 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:12.009681 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "a7ce92ff0dd04c5fb105d084915365ef"
format_stamp: "Formatted at 2025-06-26 01:54:11 on dist-test-slave-psm5"
I20250626 01:54:12.009807 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:12.036723 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:12.037236 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:12.037781 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:12.038703 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:12.038759 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.038837 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:12.038878 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.051141 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39673
I20250626 01:54:12.051164 6466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39673 every 8 connection(s)
I20250626 01:54:12.052762 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:12.055652 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:12.057193 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:12.058884 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:12.059556 413 server_base.cc:1048] running on GCE node
I20250626 01:54:12.059819 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:12.059901 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:12.059962 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902852059957 us; error 0 us; skew 500 ppm
I20250626 01:54:12.060140 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:12.060746 6467 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:12.060894 6467 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:12.061265 6467 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:12.061508 413 webserver.cc:469] Webserver started at http://127.0.103.66:38431/ using document root <none> and password file <none>
I20250626 01:54:12.061741 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:12.061826 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:12.061954 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:12.062197 6340 ts_manager.cc:194] Registered new tserver with Master: a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673)
I20250626 01:54:12.062577 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "c13696702b694623a5f7dba245ddae69"
format_stamp: "Formatted at 2025-06-26 01:54:12 on dist-test-slave-psm5"
I20250626 01:54:12.063272 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44274
I20250626 01:54:12.065104 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:54:12.066612 6479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.066993 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:12.067075 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "c13696702b694623a5f7dba245ddae69"
format_stamp: "Formatted at 2025-06-26 01:54:12 on dist-test-slave-psm5"
I20250626 01:54:12.067166 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:12.088011 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:12.088521 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:12.089000 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:12.089798 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:12.089854 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.089929 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:12.089977 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.101353 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:40385
I20250626 01:54:12.101374 6541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:40385 every 8 connection(s)
I20250626 01:54:12.103096 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:12.106024 6545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:12.106528 6546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:12.108528 6548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:12.108822 413 server_base.cc:1048] running on GCE node
I20250626 01:54:12.109059 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:12.109138 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:12.109174 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902852109169 us; error 0 us; skew 500 ppm
I20250626 01:54:12.109395 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:12.110826 6542 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:12.110961 6542 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:12.111303 6542 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:12.111725 413 webserver.cc:469] Webserver started at http://127.0.103.67:46119/ using document root <none> and password file <none>
I20250626 01:54:12.111967 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:12.112039 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:12.112178 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:12.112186 6340 ts_manager.cc:194] Registered new tserver with Master: c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:12.112807 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "887bd561c763408193cbfc84bcf5571f"
format_stamp: "Formatted at 2025-06-26 01:54:12 on dist-test-slave-psm5"
I20250626 01:54:12.112999 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44290
I20250626 01:54:12.115402 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20250626 01:54:12.117048 6553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.117383 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:12.117496 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "887bd561c763408193cbfc84bcf5571f"
format_stamp: "Formatted at 2025-06-26 01:54:12 on dist-test-slave-psm5"
I20250626 01:54:12.117602 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:12.126219 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:12.126742 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:12.127302 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:12.128198 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:12.128261 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.128348 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:12.128386 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:12.140750 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35805
I20250626 01:54:12.140774 6615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35805 every 8 connection(s)
I20250626 01:54:12.147606 6616 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:12.147763 6616 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:12.148113 6616 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:12.148900 6340 ts_manager.cc:194] Registered new tserver with Master: 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:12.149566 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.007826906s
I20250626 01:54:12.149614 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44302
I20250626 01:54:13.064544 6467 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
I20250626 01:54:13.114388 6542 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
I20250626 01:54:13.150913 6616 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
I20250626 01:54:13.178174 413 test_util.cc:276] Using random seed: -694578378
I20250626 01:54:13.194702 6340 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44306:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:54:13.195456 6340 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:54:13.214038 6431 tablet_service.cc:1468] Processing CreateTablet for tablet b8c31f23b96547d69dad82c9e3408810 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:13.214205 6432 tablet_service.cc:1468] Processing CreateTablet for tablet 878aa1c464fb40c494d2a1845e718d86 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:13.214669 6431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c31f23b96547d69dad82c9e3408810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.215365 6430 tablet_service.cc:1468] Processing CreateTablet for tablet b24acb87671143cdb8e7305523bb3668 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:13.215799 6430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24acb87671143cdb8e7305523bb3668. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.216296 6429 tablet_service.cc:1468] Processing CreateTablet for tablet 1f81c07108a848be9c387b147cd36bb6 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:13.216737 6429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f81c07108a848be9c387b147cd36bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.217188 6432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878aa1c464fb40c494d2a1845e718d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.220994 6580 tablet_service.cc:1468] Processing CreateTablet for tablet b8c31f23b96547d69dad82c9e3408810 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:13.221189 6581 tablet_service.cc:1468] Processing CreateTablet for tablet 878aa1c464fb40c494d2a1845e718d86 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:13.221421 6580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c31f23b96547d69dad82c9e3408810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.221665 6581 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878aa1c464fb40c494d2a1845e718d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.224372 6579 tablet_service.cc:1468] Processing CreateTablet for tablet b24acb87671143cdb8e7305523bb3668 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:13.224848 6579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24acb87671143cdb8e7305523bb3668. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.228415 6578 tablet_service.cc:1468] Processing CreateTablet for tablet 1f81c07108a848be9c387b147cd36bb6 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:13.229053 6578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f81c07108a848be9c387b147cd36bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.232437 6637 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:13.232954 6507 tablet_service.cc:1468] Processing CreateTablet for tablet 878aa1c464fb40c494d2a1845e718d86 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:13.233261 6636 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:13.233994 6507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878aa1c464fb40c494d2a1845e718d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.235827 6637 tablet_bootstrap.cc:654] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.237646 6636 tablet_bootstrap.cc:654] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.237915 6506 tablet_service.cc:1468] Processing CreateTablet for tablet b8c31f23b96547d69dad82c9e3408810 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:13.238394 6506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c31f23b96547d69dad82c9e3408810. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.244390 6637 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: No bootstrap required, opened a new log
I20250626 01:54:13.244747 6637 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20250626 01:54:13.245127 6505 tablet_service.cc:1468] Processing CreateTablet for tablet b24acb87671143cdb8e7305523bb3668 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:13.245731 6640 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
I20250626 01:54:13.245913 6637 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.246248 6637 raft_consensus.cc:383] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.246325 6637 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.246349 6505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b24acb87671143cdb8e7305523bb3668. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.246569 6637 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.247097 6507 tablet_service.cc:1468] Processing CreateTablet for tablet 1f81c07108a848be9c387b147cd36bb6 (DEFAULT_TABLE table=test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:13.247706 6507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f81c07108a848be9c387b147cd36bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:13.254165 6637 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20250626 01:54:13.255321 6637 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:13.257198 6637 tablet_bootstrap.cc:654] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.257339 6640 tablet_bootstrap.cc:654] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.259070 6636 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: No bootstrap required, opened a new log
I20250626 01:54:13.259271 6636 ts_tablet_manager.cc:1397] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20250626 01:54:13.260330 6636 raft_consensus.cc:357] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.260624 6636 raft_consensus.cc:383] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.260697 6636 raft_consensus.cc:738] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.260963 6636 consensus_queue.cc:260] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.262310 6640 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: No bootstrap required, opened a new log
I20250626 01:54:13.262455 6640 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250626 01:54:13.262496 6637 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: No bootstrap required, opened a new log
I20250626 01:54:13.262637 6637 ts_tablet_manager.cc:1397] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250626 01:54:13.262871 6636 ts_tablet_manager.cc:1428] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250626 01:54:13.263187 6636 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:13.263232 6640 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.263413 6640 raft_consensus.cc:383] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.263325 6637 raft_consensus.cc:357] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.263476 6640 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.263495 6637 raft_consensus.cc:383] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.263545 6637 raft_consensus.cc:738] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.263741 6640 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.263818 6637 consensus_queue.cc:260] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.265627 6636 tablet_bootstrap.cc:654] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.266180 6637 ts_tablet_manager.cc:1428] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:54:13.266566 6637 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:13.268249 6637 tablet_bootstrap.cc:654] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.270386 6637 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: No bootstrap required, opened a new log
I20250626 01:54:13.270464 6636 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: No bootstrap required, opened a new log
I20250626 01:54:13.270576 6637 ts_tablet_manager.cc:1397] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 01:54:13.270589 6636 ts_tablet_manager.cc:1397] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250626 01:54:13.271019 6640 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20250626 01:54:13.271378 6640 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
I20250626 01:54:13.271476 6637 raft_consensus.cc:357] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.271731 6637 raft_consensus.cc:383] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.271836 6637 raft_consensus.cc:738] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.272138 6637 consensus_queue.cc:260] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.272789 6637 ts_tablet_manager.cc:1428] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:13.272713 6636 raft_consensus.cc:357] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.273027 6636 raft_consensus.cc:383] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.273087 6640 tablet_bootstrap.cc:654] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.273198 6637 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:13.273226 6636 raft_consensus.cc:738] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.273597 6636 consensus_queue.cc:260] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.274434 6636 ts_tablet_manager.cc:1428] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250626 01:54:13.274874 6636 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:13.275012 6637 tablet_bootstrap.cc:654] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.277209 6636 tablet_bootstrap.cc:654] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.287181 6644 raft_consensus.cc:491] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:13.287384 6644 raft_consensus.cc:513] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.287874 6637 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: No bootstrap required, opened a new log
I20250626 01:54:13.288084 6637 ts_tablet_manager.cc:1397] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20250626 01:54:13.288286 6640 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: No bootstrap required, opened a new log
I20250626 01:54:13.288357 6644 leader_election.cc:290] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:13.288450 6640 ts_tablet_manager.cc:1397] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.012s
I20250626 01:54:13.289048 6637 raft_consensus.cc:357] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.289285 6637 raft_consensus.cc:383] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.289381 6637 raft_consensus.cc:738] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.289664 6637 consensus_queue.cc:260] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.289744 6640 raft_consensus.cc:357] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.289894 6640 raft_consensus.cc:383] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.289994 6640 raft_consensus.cc:738] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.290211 6640 consensus_queue.cc:260] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.290846 6640 ts_tablet_manager.cc:1428] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:13.292057 6640 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
I20250626 01:54:13.294268 6640 tablet_bootstrap.cc:654] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.291213 6637 ts_tablet_manager.cc:1428] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:54:13.290362 6636 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: No bootstrap required, opened a new log
I20250626 01:54:13.298161 6636 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20250626 01:54:13.299224 6636 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.299551 6636 raft_consensus.cc:383] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.299731 6636 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.299913 6640 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: No bootstrap required, opened a new log
I20250626 01:54:13.300113 6640 ts_tablet_manager.cc:1397] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250626 01:54:13.300140 6636 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.300834 6636 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:13.300971 6640 raft_consensus.cc:357] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.301177 6640 raft_consensus.cc:383] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.301370 6640 raft_consensus.cc:738] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.301625 6640 consensus_queue.cc:260] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.301884 6517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69" is_pre_election: true
I20250626 01:54:13.302156 6517 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 0.
I20250626 01:54:13.302212 6640 ts_tablet_manager.cc:1428] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:54:13.302984 6640 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
W20250626 01:54:13.303580 6468 tablet.cc:2378] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250626 01:54:13.304435 6406 leader_election.cc:304] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:13.304723 6644 raft_consensus.cc:2802] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:13.301271 6636 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:13.304838 6644 raft_consensus.cc:491] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:13.304993 6644 raft_consensus.cc:3058] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.306473 6636 tablet_bootstrap.cc:654] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.304836 6640 tablet_bootstrap.cc:654] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:13.308321 6591 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:13.308576 6591 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 0.
I20250626 01:54:13.308663 6636 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: No bootstrap required, opened a new log
I20250626 01:54:13.308892 6636 ts_tablet_manager.cc:1397] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250626 01:54:13.309082 6644 raft_consensus.cc:513] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.309388 6640 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: No bootstrap required, opened a new log
I20250626 01:54:13.309533 6640 ts_tablet_manager.cc:1397] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250626 01:54:13.309744 6636 raft_consensus.cc:357] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.309967 6636 raft_consensus.cc:383] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.310061 6636 raft_consensus.cc:738] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.310209 6517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69"
I20250626 01:54:13.310390 6517 raft_consensus.cc:3058] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.310302 6636 consensus_queue.cc:260] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.310352 6640 raft_consensus.cc:357] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.310549 6640 raft_consensus.cc:383] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:13.310647 6640 raft_consensus.cc:738] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:13.310895 6640 consensus_queue.cc:260] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.311403 6636 ts_tablet_manager.cc:1428] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250626 01:54:13.312211 6640 ts_tablet_manager.cc:1428] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:54:13.313321 6517 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:13.313714 6406 leader_election.cc:304] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:13.313799 6591 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:13.314023 6591 raft_consensus.cc:3058] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.314229 6656 raft_consensus.cc:2802] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:13.314389 6656 raft_consensus.cc:695] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 LEADER]: Becoming Leader. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Running, Role: LEADER
I20250626 01:54:13.314714 6656 consensus_queue.cc:237] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.317423 6591 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:13.317806 6340 catalog_manager.cc:5582] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef reported cstate change: term changed from 0 to 1, leader changed from <none> to a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65). New cstate: current_term: 1 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:13.309741 6644 leader_election.cc:290] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 election: Requested vote from peers c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:13.473944 6645 raft_consensus.cc:491] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:13.474200 6645 raft_consensus.cc:513] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.475101 6645 leader_election.cc:290] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:13.481750 6442 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef" is_pre_election: true
I20250626 01:54:13.481750 6591 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:13.482095 6442 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 0.
I20250626 01:54:13.482098 6591 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 0.
I20250626 01:54:13.482589 6481 leader_election.cc:304] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:13.482874 6645 raft_consensus.cc:2802] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:13.482987 6645 raft_consensus.cc:491] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:13.483093 6645 raft_consensus.cc:3058] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.485190 6645 raft_consensus.cc:513] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.485843 6645 leader_election.cc:290] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 1 election: Requested vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:13.486393 6591 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:13.486392 6442 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef"
I20250626 01:54:13.486646 6591 raft_consensus.cc:3058] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.486663 6442 raft_consensus.cc:3058] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.489867 6591 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 1.
I20250626 01:54:13.489862 6442 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 1.
I20250626 01:54:13.490440 6481 leader_election.cc:304] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887bd561c763408193cbfc84bcf5571f, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:13.490773 6645 raft_consensus.cc:2802] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:13.491200 6645 raft_consensus.cc:695] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 LEADER]: Becoming Leader. State: Replica: c13696702b694623a5f7dba245ddae69, State: Running, Role: LEADER
I20250626 01:54:13.491475 6645 consensus_queue.cc:237] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.494359 6340 catalog_manager.cc:5582] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 reported cstate change: term changed from 0 to 1, leader changed from <none> to c13696702b694623a5f7dba245ddae69 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "c13696702b694623a5f7dba245ddae69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:13.541040 6641 raft_consensus.cc:491] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:13.541265 6641 raft_consensus.cc:513] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.542124 6641 leader_election.cc:290] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:13.549297 6442 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "887bd561c763408193cbfc84bcf5571f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef" is_pre_election: true
I20250626 01:54:13.549297 6517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "887bd561c763408193cbfc84bcf5571f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69" is_pre_election: true
I20250626 01:54:13.549613 6517 raft_consensus.cc:2466] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 887bd561c763408193cbfc84bcf5571f in term 0.
I20250626 01:54:13.549613 6442 raft_consensus.cc:2466] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 887bd561c763408193cbfc84bcf5571f in term 0.
I20250626 01:54:13.550033 6555 leader_election.cc:304] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887bd561c763408193cbfc84bcf5571f, a7ce92ff0dd04c5fb105d084915365ef; no voters:
I20250626 01:54:13.550284 6641 raft_consensus.cc:2802] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:13.550377 6641 raft_consensus.cc:491] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:13.550451 6641 raft_consensus.cc:3058] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.552505 6641 raft_consensus.cc:513] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.553107 6641 leader_election.cc:290] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [CANDIDATE]: Term 1 election: Requested vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:13.553668 6517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "887bd561c763408193cbfc84bcf5571f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69"
I20250626 01:54:13.553668 6442 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "887bd561c763408193cbfc84bcf5571f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef"
I20250626 01:54:13.553934 6442 raft_consensus.cc:3058] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.553943 6517 raft_consensus.cc:3058] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.557255 6442 raft_consensus.cc:2466] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 887bd561c763408193cbfc84bcf5571f in term 1.
I20250626 01:54:13.557255 6517 raft_consensus.cc:2466] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 887bd561c763408193cbfc84bcf5571f in term 1.
I20250626 01:54:13.557848 6555 leader_election.cc:304] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887bd561c763408193cbfc84bcf5571f, a7ce92ff0dd04c5fb105d084915365ef; no voters:
I20250626 01:54:13.558175 6641 raft_consensus.cc:2802] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:13.558642 6641 raft_consensus.cc:695] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 LEADER]: Becoming Leader. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Running, Role: LEADER
I20250626 01:54:13.558972 6641 consensus_queue.cc:237] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:13.561754 6340 catalog_manager.cc:5582] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f reported cstate change: term changed from 0 to 1, leader changed from <none> to 887bd561c763408193cbfc84bcf5571f (127.0.103.67). New cstate: current_term: 1 leader_uuid: "887bd561c763408193cbfc84bcf5571f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: HEALTHY } } }
I20250626 01:54:13.583308 6644 raft_consensus.cc:491] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:13.583527 6644 raft_consensus.cc:513] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.584231 6644 leader_election.cc:290] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805), c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:13.584851 6591 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:13.584853 6517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69" is_pre_election: true
I20250626 01:54:13.585132 6591 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 0.
I20250626 01:54:13.585142 6517 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 0.
I20250626 01:54:13.585530 6406 leader_election.cc:304] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:13.585839 6644 raft_consensus.cc:2802] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:13.585932 6644 raft_consensus.cc:491] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:13.586026 6644 raft_consensus.cc:3058] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.588050 6644 raft_consensus.cc:513] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.588635 6644 leader_election.cc:290] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 election: Requested vote from peers 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805), c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:13.589223 6591 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:13.589336 6517 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69"
I20250626 01:54:13.589412 6591 raft_consensus.cc:3058] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.589545 6517 raft_consensus.cc:3058] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:13.591500 6591 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:13.591598 6517 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:13.592036 6406 leader_election.cc:304] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:13.592355 6644 raft_consensus.cc:2802] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:13.592499 6644 raft_consensus.cc:695] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 LEADER]: Becoming Leader. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Running, Role: LEADER
I20250626 01:54:13.592769 6644 consensus_queue.cc:237] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:13.595721 6340 catalog_manager.cc:5582] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef reported cstate change: term changed from 0 to 1, leader changed from <none> to a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65). New cstate: current_term: 1 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:13.673754 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:54:13.683871 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:13.684378 413 tablet_replica.cc:331] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: stopping tablet replica
I20250626 01:54:13.684607 413 raft_consensus.cc:2241] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.684808 413 raft_consensus.cc:2270] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.685535 413 tablet_replica.cc:331] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: stopping tablet replica
I20250626 01:54:13.685684 413 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:13.685937 413 pending_rounds.cc:70] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Trying to abort 1 pending ops.
I20250626 01:54:13.685998 413 pending_rounds.cc:77] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171698088318443520 op_type: NO_OP noop_request { }
I20250626 01:54:13.686120 413 raft_consensus.cc:2887] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 01:54:13.686211 413 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.687038 413 tablet_replica.cc:331] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: stopping tablet replica
I20250626 01:54:13.687184 413 raft_consensus.cc:2241] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:13.687384 413 pending_rounds.cc:70] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Trying to abort 1 pending ops.
I20250626 01:54:13.687440 413 pending_rounds.cc:77] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171698087181758464 op_type: NO_OP noop_request { }
I20250626 01:54:13.687539 413 raft_consensus.cc:2887] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 01:54:13.687633 413 raft_consensus.cc:2270] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.688459 413 tablet_replica.cc:331] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: stopping tablet replica
I20250626 01:54:13.688597 413 raft_consensus.cc:2241] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.688730 413 raft_consensus.cc:2270] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.705155 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20250626 01:54:13.708986 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:54:13.728552 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:13.728938 413 tablet_replica.cc:331] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: stopping tablet replica
I20250626 01:54:13.729140 413 raft_consensus.cc:2241] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.729312 413 raft_consensus.cc:2270] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.729996 413 tablet_replica.cc:331] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: stopping tablet replica
I20250626 01:54:13.730139 413 raft_consensus.cc:2241] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.730255 413 raft_consensus.cc:2270] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.730755 413 tablet_replica.cc:331] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: stopping tablet replica
I20250626 01:54:13.730854 413 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.730958 413 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.731447 413 tablet_replica.cc:331] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: stopping tablet replica
I20250626 01:54:13.731552 413 raft_consensus.cc:2241] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:13.731737 413 pending_rounds.cc:70] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Trying to abort 1 pending ops.
I20250626 01:54:13.731775 413 pending_rounds.cc:77] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171698087903715328 op_type: NO_OP noop_request { }
I20250626 01:54:13.731876 413 raft_consensus.cc:2887] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 01:54:13.732020 413 raft_consensus.cc:2270] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.746906 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:54:13.750705 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:54:13.759805 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:13.760274 413 tablet_replica.cc:331] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: stopping tablet replica
I20250626 01:54:13.760500 413 raft_consensus.cc:2241] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.760675 413 raft_consensus.cc:2270] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.761353 413 tablet_replica.cc:331] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: stopping tablet replica
I20250626 01:54:13.761474 413 raft_consensus.cc:2241] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.761590 413 raft_consensus.cc:2270] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.762120 413 tablet_replica.cc:331] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: stopping tablet replica
I20250626 01:54:13.762231 413 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:13.762338 413 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.762749 413 tablet_replica.cc:331] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: stopping tablet replica
I20250626 01:54:13.762866 413 raft_consensus.cc:2241] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:13.763057 413 pending_rounds.cc:70] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Trying to abort 1 pending ops.
I20250626 01:54:13.763098 413 pending_rounds.cc:77] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7171698088180260864 op_type: NO_OP noop_request { }
I20250626 01:54:13.763192 413 raft_consensus.cc:2887] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250626 01:54:13.763259 413 raft_consensus.cc:2270] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:13.778184 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:54:13.782373 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:13.785071 6668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:13.785296 6669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:13.785614 6671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:13.786564 413 server_base.cc:1048] running on GCE node
I20250626 01:54:13.786839 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:13.786923 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:13.786958 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902853786956 us; error 0 us; skew 500 ppm
I20250626 01:54:13.787104 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:13.788461 413 webserver.cc:469] Webserver started at http://127.0.103.68:46145/ using document root <none> and password file <none>
I20250626 01:54:13.788699 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:13.788774 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:13.788897 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:13.789522 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "c692a6c1cc364c488575df1944e04351"
format_stamp: "Formatted at 2025-06-26 01:54:13 on dist-test-slave-psm5"
I20250626 01:54:13.792357 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250626 01:54:13.794124 6676 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:13.794520 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:54:13.794649 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "c692a6c1cc364c488575df1944e04351"
format_stamp: "Formatted at 2025-06-26 01:54:13 on dist-test-slave-psm5"
I20250626 01:54:13.794764 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:13.804162 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:13.804739 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:13.805315 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:13.806710 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:13.806795 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:13.806896 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:13.806968 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:13.819221 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:34489
I20250626 01:54:13.819233 6738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:34489 every 8 connection(s)
I20250626 01:54:13.826725 6739 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:13.826897 6739 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:13.827239 6739 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:13.828131 6340 ts_manager.cc:194] Registered new tserver with Master: c692a6c1cc364c488575df1944e04351 (127.0.103.68:34489)
I20250626 01:54:13.829028 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44308
W20250626 01:54:13.901129 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:13.902578 6325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20250626 01:54:13.904089 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:13.904212 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:13.906213 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:13.906332 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:13.908035 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:13.908140 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
I20250626 01:54:14.830382 6739 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
W20250626 01:54:14.912097 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:14.921309 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:14.921430 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:14.923372 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:14.923475 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:14.925340 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:14.925433 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:15.929221 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:15.931353 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:15.931458 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:15.932833 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:15.932912 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:15.934172 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:15.934245 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:16.938176 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:16.940914 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:16.941020 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:16.942589 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:16.942691 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:16.944121 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:16.944204 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:40385: connect: Connection refused (error 111)
W20250626 01:54:17.948037 6388 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:17.951771 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:17.951872 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:35805: connect: Connection refused (error 111)
W20250626 01:54:17.953513 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:17.953609 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:17.955266 6388 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:17.955350 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39673: connect: Connection refused (error 111)
W20250626 01:54:18.956365 6388 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver a7ce92ff0dd04c5fb105d084915365ef not available for placement
I20250626 01:54:18.958019 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:18.961004 6758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:18.960994 6759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:18.961879 6761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:18.973706 413 server_base.cc:1048] running on GCE node
I20250626 01:54:18.975427 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:18.975512 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:18.975559 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902858975535 us; error 0 us; skew 500 ppm
I20250626 01:54:18.975741 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:18.977361 413 webserver.cc:469] Webserver started at http://127.0.103.65:36213/ using document root <none> and password file <none>
I20250626 01:54:18.977615 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:18.977701 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 01:54:18.979773 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:18.981635 6766 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:18.982053 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20250626 01:54:18.982177 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "a7ce92ff0dd04c5fb105d084915365ef"
format_stamp: "Formatted at 2025-06-26 01:54:11 on dist-test-slave-psm5"
I20250626 01:54:18.982321 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:18.993041 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:18.993618 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:18.994230 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:18.995735 6773 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20250626 01:54:19.007757 413 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250626 01:54:19.007952 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.000s
I20250626 01:54:19.008111 413 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250626 01:54:19.011003 6773 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:19.015707 6773 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.016178 6773 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap complete.
I20250626 01:54:19.016453 6773 ts_tablet_manager.cc:1397] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20250626 01:54:19.017215 6773 raft_consensus.cc:357] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.017408 6773 raft_consensus.cc:738] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.017704 6773 consensus_queue.cc:260] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.018600 6773 ts_tablet_manager.cc:1428] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:19.018954 6773 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:19.019641 413 ts_tablet_manager.cc:610] Registered 4 tablets
I20250626 01:54:19.019737 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.005s sys 0.003s
I20250626 01:54:19.024080 6773 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 01:54:19.024689 6773 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap complete.
I20250626 01:54:19.024959 6773 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20250626 01:54:19.025923 6773 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:19.026300 6773 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.026640 6773 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:19.027312 6773 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250626 01:54:19.027617 6773 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:19.031533 6773 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.031980 6773 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap complete.
I20250626 01:54:19.032250 6773 ts_tablet_manager.cc:1397] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250626 01:54:19.033193 6773 raft_consensus.cc:357] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.033412 6773 raft_consensus.cc:738] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.033749 6773 consensus_queue.cc:260] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.034338 6773 ts_tablet_manager.cc:1428] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:19.034618 6773 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap starting.
I20250626 01:54:19.039044 6773 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 01:54:19.039489 6773 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Bootstrap complete.
I20250626 01:54:19.039812 6773 ts_tablet_manager.cc:1397] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250626 01:54:19.040573 6773 raft_consensus.cc:357] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.040839 6773 raft_consensus.cc:738] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.041137 6773 consensus_queue.cc:260] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.041714 6773 ts_tablet_manager.cc:1428] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:54:19.042627 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39673
I20250626 01:54:19.042629 6834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39673 every 8 connection(s)
I20250626 01:54:19.044160 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:19.047439 6839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:19.049528 6840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:19.050643 6842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:19.050738 413 server_base.cc:1048] running on GCE node
I20250626 01:54:19.051287 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:19.051394 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:19.051465 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902859051440 us; error 0 us; skew 500 ppm
I20250626 01:54:19.051633 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:19.052969 6835 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:19.053153 6835 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:19.053176 413 webserver.cc:469] Webserver started at http://127.0.103.66:38431/ using document root <none> and password file <none>
I20250626 01:54:19.053424 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:19.053557 6835 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:19.055087 6340 ts_manager.cc:194] Re-registered known tserver with Master: a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673)
I20250626 01:54:19.057078 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57550
I20250626 01:54:19.066504 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 01:54:19.068523 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.004s sys 0.000s
I20250626 01:54:19.070042 6847 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:19.070374 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:54:19.070478 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "c13696702b694623a5f7dba245ddae69"
format_stamp: "Formatted at 2025-06-26 01:54:12 on dist-test-slave-psm5"
I20250626 01:54:19.070596 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:19.105525 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:19.106144 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:19.106685 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:19.115334 413 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250626 01:54:19.115458 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20250626 01:54:19.117472 6854 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
I20250626 01:54:19.120899 6854 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.121399 6854 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Bootstrap complete.
I20250626 01:54:19.121658 6854 ts_tablet_manager.cc:1397] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250626 01:54:19.121996 413 ts_tablet_manager.cc:610] Registered 4 tablets
I20250626 01:54:19.122057 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.002s sys 0.003s
I20250626 01:54:19.122225 6854 raft_consensus.cc:357] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.122421 6854 raft_consensus.cc:738] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.122725 6854 consensus_queue.cc:260] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.123420 6854 ts_tablet_manager.cc:1428] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20250626 01:54:19.123660 6854 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
I20250626 01:54:19.127857 6854 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.128368 6854 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Bootstrap complete.
I20250626 01:54:19.128713 6854 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20250626 01:54:19.129673 6854 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:19.129925 6854 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.130270 6854 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:19.130923 6854 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:19.131268 6854 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
I20250626 01:54:19.137616 6854 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 01:54:19.138026 6854 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Bootstrap complete.
I20250626 01:54:19.138154 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:40385
I20250626 01:54:19.138155 6914 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:40385 every 8 connection(s)
I20250626 01:54:19.138280 6854 ts_tablet_manager.cc:1397] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250626 01:54:19.138885 6854 raft_consensus.cc:357] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.139117 6854 raft_consensus.cc:738] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.139391 6854 consensus_queue.cc:260] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.139858 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:54:19.140062 6854 ts_tablet_manager.cc:1428] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:54:19.140333 6854 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Bootstrap starting.
W20250626 01:54:19.143360 6918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:19.152205 6919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:19.153520 6854 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.153609 6915 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:19.153716 6915 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:19.153919 6854 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Bootstrap complete.
I20250626 01:54:19.154143 6915 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:19.154193 6854 ts_tablet_manager.cc:1397] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250626 01:54:19.154964 6854 raft_consensus.cc:357] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.155189 6854 raft_consensus.cc:738] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c13696702b694623a5f7dba245ddae69, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.155529 6854 consensus_queue.cc:260] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.156153 6854 ts_tablet_manager.cc:1428] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250626 01:54:19.156242 6340 ts_manager.cc:194] Re-registered known tserver with Master: c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:19.156409 413 server_base.cc:1048] running on GCE node
W20250626 01:54:19.156483 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:19.156848 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:19.156889 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:19.156924 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902859156909 us; error 0 us; skew 500 ppm
I20250626 01:54:19.157018 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:19.158509 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57554
I20250626 01:54:19.158509 413 webserver.cc:469] Webserver started at http://127.0.103.67:46119/ using document root <none> and password file <none>
I20250626 01:54:19.158761 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:19.158798 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 01:54:19.159795 6915 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
I20250626 01:54:19.160907 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:19.162561 6927 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:19.162935 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:54:19.163023 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "887bd561c763408193cbfc84bcf5571f"
format_stamp: "Formatted at 2025-06-26 01:54:12 on dist-test-slave-psm5"
I20250626 01:54:19.163139 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:19.178427 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:19.179018 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:19.179642 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:19.190541 413 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250626 01:54:19.190701 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20250626 01:54:19.194351 6934 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:19.199854 6934 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250626 01:54:19.200292 6934 tablet_bootstrap.cc:492] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Bootstrap complete.
I20250626 01:54:19.200361 413 ts_tablet_manager.cc:610] Registered 4 tablets
I20250626 01:54:19.200445 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.009s sys 0.000s
I20250626 01:54:19.200551 6934 ts_tablet_manager.cc:1397] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20250626 01:54:19.201325 6934 raft_consensus.cc:357] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.201555 6934 raft_consensus.cc:738] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.201774 6934 consensus_queue.cc:260] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.202519 6934 ts_tablet_manager.cc:1428] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:19.202823 6934 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:19.207020 6934 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.207396 6934 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Bootstrap complete.
I20250626 01:54:19.207623 6934 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250626 01:54:19.208501 6934 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:19.208673 6934 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.209020 6934 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:19.209594 6934 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:19.209831 6934 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:19.214300 6934 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.214746 6934 tablet_bootstrap.cc:492] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Bootstrap complete.
I20250626 01:54:19.215013 6934 ts_tablet_manager.cc:1397] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250626 01:54:19.215999 6934 raft_consensus.cc:357] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.216195 6934 raft_consensus.cc:738] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.216488 6934 consensus_queue.cc:260] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.217432 6934 ts_tablet_manager.cc:1428] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:19.217844 6934 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Bootstrap starting.
I20250626 01:54:19.223253 6934 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:19.223776 6934 tablet_bootstrap.cc:492] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Bootstrap complete.
I20250626 01:54:19.224074 6934 ts_tablet_manager.cc:1397] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20250626 01:54:19.224799 6934 raft_consensus.cc:357] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.225083 6934 raft_consensus.cc:738] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 887bd561c763408193cbfc84bcf5571f, State: Initialized, Role: FOLLOWER
I20250626 01:54:19.225180 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35805
I20250626 01:54:19.225221 6995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35805 every 8 connection(s)
I20250626 01:54:19.225427 6934 consensus_queue.cc:260] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:19.225927 6934 ts_tablet_manager.cc:1428] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:19.233003 6996 heartbeater.cc:344] Connected to a master server at 127.0.103.126:40787
I20250626 01:54:19.233183 6996 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:19.233505 6996 heartbeater.cc:507] Master 127.0.103.126:40787 requested a full tablet report, sending...
I20250626 01:54:19.234830 6340 ts_manager.cc:194] Re-registered known tserver with Master: 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:19.236569 6340 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57560
I20250626 01:54:19.237779 6996 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
W20250626 01:54:19.966287 6388 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica c13696702b694623a5f7dba245ddae69 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } }
W20250626 01:54:19.968081 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b24acb87671143cdb8e7305523bb3668, TS a7ce92ff0dd04c5fb105d084915365ef -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:19.968199 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b24acb87671143cdb8e7305523bb3668, TS a7ce92ff0dd04c5fb105d084915365ef -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:19.975162 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b8c31f23b96547d69dad82c9e3408810, TS c13696702b694623a5f7dba245ddae69 -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:19.975291 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b8c31f23b96547d69dad82c9e3408810, TS c13696702b694623a5f7dba245ddae69 -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:19.982627 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1f81c07108a848be9c387b147cd36bb6, TS 887bd561c763408193cbfc84bcf5571f -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:19.982759 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1f81c07108a848be9c387b147cd36bb6, TS 887bd561c763408193cbfc84bcf5571f -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.058830 6835 heartbeater.cc:499] Master 127.0.103.126:40787 was elected leader, sending a full tablet report...
I20250626 01:54:20.277209 7003 raft_consensus.cc:491] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:20.277397 7003 raft_consensus.cc:513] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:20.278151 7003 leader_election.cc:290] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805), c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:20.284834 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:20.284896 6889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69" is_pre_election: true
I20250626 01:54:20.285072 6889 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:20.285072 6970 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:20.285534 6768 leader_election.cc:304] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:20.285784 7003 raft_consensus.cc:2802] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:54:20.285871 7003 raft_consensus.cc:491] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:20.285959 7003 raft_consensus.cc:3058] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.287966 7003 raft_consensus.cc:513] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:20.288558 7003 leader_election.cc:290] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 election: Requested vote from peers 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805), c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385)
I20250626 01:54:20.289049 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:20.289083 6889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "878aa1c464fb40c494d2a1845e718d86" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69"
I20250626 01:54:20.289181 6970 raft_consensus.cc:3058] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.289242 6889 raft_consensus.cc:3058] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.291416 6970 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 2.
I20250626 01:54:20.291416 6889 raft_consensus.cc:2466] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 2.
I20250626 01:54:20.291860 6768 leader_election.cc:304] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887bd561c763408193cbfc84bcf5571f, a7ce92ff0dd04c5fb105d084915365ef; no voters:
I20250626 01:54:20.292186 7003 raft_consensus.cc:2802] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:54:20.292582 7003 raft_consensus.cc:695] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Becoming Leader. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Running, Role: LEADER
I20250626 01:54:20.292830 7003 consensus_queue.cc:237] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } }
I20250626 01:54:20.295394 6340 catalog_manager.cc:5582] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:20.311460 7012 raft_consensus.cc:491] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:20.311638 7012 raft_consensus.cc:513] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.312505 7012 leader_election.cc:290] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:20.318552 6809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef" is_pre_election: true
I20250626 01:54:20.318796 6809 raft_consensus.cc:2466] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 1.
I20250626 01:54:20.319234 6849 leader_election.cc:304] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:20.319536 7012 raft_consensus.cc:2802] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:54:20.319623 7012 raft_consensus.cc:491] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:20.319612 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:20.319746 7012 raft_consensus.cc:3058] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.319926 6970 raft_consensus.cc:2408] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c13696702b694623a5f7dba245ddae69 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 01:54:20.321924 7012 raft_consensus.cc:513] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.322521 7012 leader_election.cc:290] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 election: Requested vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:20.323040 6809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef"
I20250626 01:54:20.323096 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8c31f23b96547d69dad82c9e3408810" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:20.323197 6809 raft_consensus.cc:3058] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.323235 6970 raft_consensus.cc:3058] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.325381 6809 raft_consensus.cc:2466] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 2.
I20250626 01:54:20.325456 6970 raft_consensus.cc:2408] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c13696702b694623a5f7dba245ddae69 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250626 01:54:20.325930 6849 leader_election.cc:304] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters: 887bd561c763408193cbfc84bcf5571f
I20250626 01:54:20.326207 7012 raft_consensus.cc:2802] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:54:20.326601 7012 raft_consensus.cc:695] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 2 LEADER]: Becoming Leader. State: Replica: c13696702b694623a5f7dba245ddae69, State: Running, Role: LEADER
I20250626 01:54:20.326900 7012 consensus_queue.cc:237] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.329515 6340 catalog_manager.cc:5582] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 reported cstate change: term changed from 1 to 2, leader changed from 887bd561c763408193cbfc84bcf5571f (127.0.103.67) to c13696702b694623a5f7dba245ddae69 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "c13696702b694623a5f7dba245ddae69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:20.583282 7012 raft_consensus.cc:491] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:20.583542 7012 raft_consensus.cc:513] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.584271 7012 leader_election.cc:290] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:20.584875 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:20.584875 6809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef" is_pre_election: true
I20250626 01:54:20.585129 6970 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 1.
I20250626 01:54:20.585129 6809 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 1.
I20250626 01:54:20.585551 6849 leader_election.cc:304] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887bd561c763408193cbfc84bcf5571f, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:20.585788 7012 raft_consensus.cc:2802] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:54:20.585872 7012 raft_consensus.cc:491] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:20.585940 7012 raft_consensus.cc:3058] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.588069 7012 raft_consensus.cc:513] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.588624 7012 leader_election.cc:290] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 election: Requested vote from peers a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:20.589149 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:20.589149 6809 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b24acb87671143cdb8e7305523bb3668" candidate_uuid: "c13696702b694623a5f7dba245ddae69" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a7ce92ff0dd04c5fb105d084915365ef"
I20250626 01:54:20.589344 6809 raft_consensus.cc:3058] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.589344 6970 raft_consensus.cc:3058] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.592451 6809 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 2.
I20250626 01:54:20.592451 6970 raft_consensus.cc:2466] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c13696702b694623a5f7dba245ddae69 in term 2.
I20250626 01:54:20.593030 6849 leader_election.cc:304] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:20.593336 7012 raft_consensus.cc:2802] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:54:20.593473 7012 raft_consensus.cc:695] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 2 LEADER]: Becoming Leader. State: Replica: c13696702b694623a5f7dba245ddae69, State: Running, Role: LEADER
I20250626 01:54:20.593784 7012 consensus_queue.cc:237] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.596412 6340 catalog_manager.cc:5582] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "c13696702b694623a5f7dba245ddae69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:20.722676 6889 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:54:20.723284 7003 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:20.729275 6970 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:54:20.729967 7003 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:20.733980 6809 raft_consensus.cc:1273] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Refusing update from remote peer c13696702b694623a5f7dba245ddae69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250626 01:54:20.734509 7012 consensus_queue.cc:1035] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:20.738272 6970 raft_consensus.cc:1273] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer c13696702b694623a5f7dba245ddae69: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250626 01:54:20.738449 6970 pending_rounds.cc:85] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Aborting all ops after (but not including) 0
I20250626 01:54:20.738531 6970 pending_rounds.cc:107] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250626 01:54:20.738579 6970 raft_consensus.cc:2887] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250626 01:54:20.739257 7012 consensus_queue.cc:1035] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:20.823799 7003 raft_consensus.cc:491] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:20.824048 7003 raft_consensus.cc:513] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.824633 7003 leader_election.cc:290] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:20.825244 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f" is_pre_election: true
I20250626 01:54:20.825244 6889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69" is_pre_election: true
I20250626 01:54:20.825495 6970 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:20.825495 6889 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 1.
I20250626 01:54:20.825907 6768 leader_election.cc:304] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7ce92ff0dd04c5fb105d084915365ef, c13696702b694623a5f7dba245ddae69; no voters:
I20250626 01:54:20.826133 7003 raft_consensus.cc:2802] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250626 01:54:20.826215 7003 raft_consensus.cc:491] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:20.826282 7003 raft_consensus.cc:3058] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.828502 7003 raft_consensus.cc:513] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.829104 7003 leader_election.cc:290] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 election: Requested vote from peers c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385), 887bd561c763408193cbfc84bcf5571f (127.0.103.67:35805)
I20250626 01:54:20.829612 6970 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "887bd561c763408193cbfc84bcf5571f"
I20250626 01:54:20.829675 6889 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f81c07108a848be9c387b147cd36bb6" candidate_uuid: "a7ce92ff0dd04c5fb105d084915365ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c13696702b694623a5f7dba245ddae69"
I20250626 01:54:20.829794 6889 raft_consensus.cc:3058] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.829794 6970 raft_consensus.cc:3058] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 1 FOLLOWER]: Advancing to term 2
I20250626 01:54:20.832893 6889 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 2.
I20250626 01:54:20.832893 6970 raft_consensus.cc:2466] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7ce92ff0dd04c5fb105d084915365ef in term 2.
I20250626 01:54:20.833482 6768 leader_election.cc:304] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 887bd561c763408193cbfc84bcf5571f, a7ce92ff0dd04c5fb105d084915365ef; no voters:
I20250626 01:54:20.833739 7003 raft_consensus.cc:2802] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Leader election won for term 2
I20250626 01:54:20.833873 7003 raft_consensus.cc:695] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Becoming Leader. State: Replica: a7ce92ff0dd04c5fb105d084915365ef, State: Running, Role: LEADER
I20250626 01:54:20.834142 7003 consensus_queue.cc:237] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:20.836687 6340 catalog_manager.cc:5582] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:20.987478 6809 consensus_queue.cc:237] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } }
I20250626 01:54:20.990087 6889 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:54:20.989998 6970 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250626 01:54:20.991112 7003 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:20.991467 7006 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:20.993942 7021 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c692a6c1cc364c488575df1944e04351 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } } }
I20250626 01:54:20.995222 6970 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c692a6c1cc364c488575df1944e04351 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } } }
I20250626 01:54:20.995234 6889 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c692a6c1cc364c488575df1944e04351 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } } }
W20250626 01:54:21.000100 6769 consensus_peers.cc:487] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef -> Peer c692a6c1cc364c488575df1944e04351 (127.0.103.68:34489): Couldn't send request to peer c692a6c1cc364c488575df1944e04351. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 878aa1c464fb40c494d2a1845e718d86. This is attempt 1: this message will repeat every 5th retry.
W20250626 01:54:21.000434 6388 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20250626 01:54:21.002171 6340 catalog_manager.cc:5582] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef reported cstate change: config changed from index -1 to 3, NON_VOTER c692a6c1cc364c488575df1944e04351 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250626 01:54:21.003924 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b24acb87671143cdb8e7305523bb3668, TS a7ce92ff0dd04c5fb105d084915365ef -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:21.004051 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b24acb87671143cdb8e7305523bb3668, TS a7ce92ff0dd04c5fb105d084915365ef -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:21.008431 6388 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b8c31f23b96547d69dad82c9e3408810, TS 887bd561c763408193cbfc84bcf5571f -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
W20250626 01:54:21.008564 6388 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b8c31f23b96547d69dad82c9e3408810, TS 887bd561c763408193cbfc84bcf5571f -> TS c692a6c1cc364c488575df1944e04351 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } }
I20250626 01:54:21.107722 6970 raft_consensus.cc:1273] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer c13696702b694623a5f7dba245ddae69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:54:21.108376 7012 consensus_queue.cc:1035] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:21.112720 6808 raft_consensus.cc:1273] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Refusing update from remote peer c13696702b694623a5f7dba245ddae69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:54:21.113651 7026 consensus_queue.cc:1035] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:21.242194 6970 raft_consensus.cc:1273] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:54:21.242723 7021 consensus_queue.cc:1035] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:21.247429 6889 raft_consensus.cc:1273] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250626 01:54:21.248358 7003 consensus_queue.cc:1035] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:21.471628 7037 ts_tablet_manager.cc:927] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: Initiating tablet copy from peer a7ce92ff0dd04c5fb105d084915365ef (127.0.103.65:39673)
I20250626 01:54:21.472590 7037 tablet_copy_client.cc:323] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:39673
I20250626 01:54:21.484580 6820 tablet_copy_service.cc:140] P a7ce92ff0dd04c5fb105d084915365ef: Received BeginTabletCopySession request for tablet 878aa1c464fb40c494d2a1845e718d86 from peer c692a6c1cc364c488575df1944e04351 ({username='slave'} at 127.0.0.1:40968)
I20250626 01:54:21.484778 6820 tablet_copy_service.cc:161] P a7ce92ff0dd04c5fb105d084915365ef: Beginning new tablet copy session on tablet 878aa1c464fb40c494d2a1845e718d86 from peer c692a6c1cc364c488575df1944e04351 at {username='slave'} at 127.0.0.1:40968: session id = c692a6c1cc364c488575df1944e04351-878aa1c464fb40c494d2a1845e718d86
I20250626 01:54:21.487531 6820 tablet_copy_source_session.cc:215] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:21.488665 7037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 878aa1c464fb40c494d2a1845e718d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:21.492997 7037 tablet_copy_client.cc:806] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:21.493311 7037 tablet_copy_client.cc:670] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:21.494752 7037 tablet_copy_client.cc:538] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:21.498143 7037 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: Bootstrap starting.
I20250626 01:54:21.505044 7037 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:21.505457 7037 tablet_bootstrap.cc:492] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: Bootstrap complete.
I20250626 01:54:21.505725 7037 ts_tablet_manager.cc:1397] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20250626 01:54:21.506500 7037 raft_consensus.cc:357] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } }
I20250626 01:54:21.506683 7037 raft_consensus.cc:738] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c692a6c1cc364c488575df1944e04351, State: Initialized, Role: LEARNER
I20250626 01:54:21.506848 7037 consensus_queue.cc:260] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: true } }
I20250626 01:54:21.507588 7037 ts_tablet_manager.cc:1428] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:54:21.508180 6820 tablet_copy_service.cc:342] P a7ce92ff0dd04c5fb105d084915365ef: Request end of tablet copy session c692a6c1cc364c488575df1944e04351-878aa1c464fb40c494d2a1845e718d86 received from {username='slave'} at 127.0.0.1:40968
I20250626 01:54:21.508308 6820 tablet_copy_service.cc:434] P a7ce92ff0dd04c5fb105d084915365ef: ending tablet copy session c692a6c1cc364c488575df1944e04351-878aa1c464fb40c494d2a1845e718d86 on tablet 878aa1c464fb40c494d2a1845e718d86 with peer c692a6c1cc364c488575df1944e04351
I20250626 01:54:22.052685 6714 raft_consensus.cc:1215] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250626 01:54:22.530702 7003 raft_consensus.cc:1062] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef: attempting to promote NON_VOTER c692a6c1cc364c488575df1944e04351 to VOTER
I20250626 01:54:22.531563 7003 consensus_queue.cc:237] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } }
I20250626 01:54:22.533859 6970 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:54:22.533939 6714 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 LEARNER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:54:22.534299 6889 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250626 01:54:22.534407 7045 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:22.534823 7003 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:22.535140 7021 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:22.538134 7045 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c692a6c1cc364c488575df1944e04351 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.539079 6714 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c692a6c1cc364c488575df1944e04351 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.539108 6970 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c692a6c1cc364c488575df1944e04351 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.543367 6338 catalog_manager.cc:5582] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 reported cstate change: config changed from index 3 to 4, c692a6c1cc364c488575df1944e04351 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.546130 6889 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, c692a6c1cc364c488575df1944e04351 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c13696702b694623a5f7dba245ddae69" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 40385 } attrs { replace: true } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.564251 6808 consensus_queue.cc:237] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } }
I20250626 01:54:22.566150 6714 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 01:54:22.566309 6970 raft_consensus.cc:1273] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Refusing update from remote peer a7ce92ff0dd04c5fb105d084915365ef: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250626 01:54:22.567737 7003 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:22.568310 7003 consensus_queue.cc:1035] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:22.570168 7003 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c13696702b694623a5f7dba245ddae69 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.570786 6970 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c13696702b694623a5f7dba245ddae69 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.573555 6383 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 878aa1c464fb40c494d2a1845e718d86 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:22.575161 6337 catalog_manager.cc:5582] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef reported cstate change: config changed from index 4 to 5, VOTER c13696702b694623a5f7dba245ddae69 (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "a7ce92ff0dd04c5fb105d084915365ef" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:22.575636 6714 raft_consensus.cc:2953] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER c13696702b694623a5f7dba245ddae69 (127.0.103.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7ce92ff0dd04c5fb105d084915365ef" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39673 } } peers { permanent_uuid: "887bd561c763408193cbfc84bcf5571f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35805 } } peers { permanent_uuid: "c692a6c1cc364c488575df1944e04351" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34489 } attrs { promote: false } } }
I20250626 01:54:22.585819 6879 tablet_service.cc:1515] Processing DeleteTablet for tablet 878aa1c464fb40c494d2a1845e718d86 with delete_type TABLET_DATA_TOMBSTONED (TS c13696702b694623a5f7dba245ddae69 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40084
I20250626 01:54:22.586503 7056 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:22.586687 7056 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:22.586938 7056 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:22.587828 7056 ts_tablet_manager.cc:1905] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:22.602084 7056 ts_tablet_manager.cc:1918] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250626 01:54:22.602212 7056 log.cc:1199] T 878aa1c464fb40c494d2a1845e718d86 P c13696702b694623a5f7dba245ddae69: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1750902684575999-413-0/minicluster-data/ts-1-root/wals/878aa1c464fb40c494d2a1845e718d86
I20250626 01:54:22.602993 6325 catalog_manager.cc:4928] TS c13696702b694623a5f7dba245ddae69 (127.0.103.66:40385): tablet 878aa1c464fb40c494d2a1845e718d86 (table test-workload [id=b8d07bc6b0e64b369bad99cc5e5010ca]) successfully deleted
I20250626 01:54:23.253764 413 tablet_server.cc:178] TabletServer@127.0.103.65:39673 shutting down...
I20250626 01:54:23.262550 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:23.262850 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.263051 413 raft_consensus.cc:2241] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:54:23.263392 413 raft_consensus.cc:2270] T 1f81c07108a848be9c387b147cd36bb6 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.264092 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.264245 413 raft_consensus.cc:2241] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.264408 413 raft_consensus.cc:2270] T b24acb87671143cdb8e7305523bb3668 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.264923 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.265049 413 raft_consensus.cc:2241] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.265189 413 raft_consensus.cc:2270] T b8c31f23b96547d69dad82c9e3408810 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.265609 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.265717 413 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:54:23.265995 413 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P a7ce92ff0dd04c5fb105d084915365ef [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.280817 413 tablet_server.cc:195] TabletServer@127.0.103.65:39673 shutdown complete.
I20250626 01:54:23.284710 413 tablet_server.cc:178] TabletServer@127.0.103.66:40385 shutting down...
I20250626 01:54:23.293478 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:23.293764 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.293967 413 raft_consensus.cc:2241] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.294144 413 raft_consensus.cc:2270] T 1f81c07108a848be9c387b147cd36bb6 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.294816 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.294930 413 raft_consensus.cc:2241] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:54:23.295197 413 raft_consensus.cc:2270] T b24acb87671143cdb8e7305523bb3668 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.295696 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.295802 413 raft_consensus.cc:2241] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 2 LEADER]: Raft consensus shutting down.
I20250626 01:54:23.296128 413 raft_consensus.cc:2270] T b8c31f23b96547d69dad82c9e3408810 P c13696702b694623a5f7dba245ddae69 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.310477 413 tablet_server.cc:195] TabletServer@127.0.103.66:40385 shutdown complete.
I20250626 01:54:23.314191 413 tablet_server.cc:178] TabletServer@127.0.103.67:35805 shutting down...
I20250626 01:54:23.322706 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:23.322979 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.323172 413 raft_consensus.cc:2241] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.323338 413 raft_consensus.cc:2270] T 1f81c07108a848be9c387b147cd36bb6 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.323966 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.324084 413 raft_consensus.cc:2241] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.324196 413 raft_consensus.cc:2270] T b24acb87671143cdb8e7305523bb3668 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.324602 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.324717 413 raft_consensus.cc:2241] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.324819 413 raft_consensus.cc:2270] T b8c31f23b96547d69dad82c9e3408810 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.325196 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:23.325284 413 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.325464 413 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P 887bd561c763408193cbfc84bcf5571f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.339753 413 tablet_server.cc:195] TabletServer@127.0.103.67:35805 shutdown complete.
I20250626 01:54:23.343428 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:54:23.351778 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:23.352075 413 tablet_replica.cc:331] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351: stopping tablet replica
I20250626 01:54:23.352255 413 raft_consensus.cc:2241] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:23.352499 413 raft_consensus.cc:2270] T 878aa1c464fb40c494d2a1845e718d86 P c692a6c1cc364c488575df1944e04351 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.366353 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:54:23.369599 413 master.cc:561] Master@127.0.103.126:40787 shutting down...
I20250626 01:54:23.377323 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:23.377592 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:23.377696 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 348701fbef564bc3b08a7b4722cb0f5a: stopping tablet replica
I20250626 01:54:23.391577 413 master.cc:583] Master@127.0.103.126:40787 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11574 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250626 01:54:23.406205 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41555
I20250626 01:54:23.406771 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:23.409161 7058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.409171 7057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.409338 7060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:23.409699 413 server_base.cc:1048] running on GCE node
I20250626 01:54:23.409930 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:23.409986 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:23.410022 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902863410018 us; error 0 us; skew 500 ppm
I20250626 01:54:23.410179 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:23.414827 413 webserver.cc:469] Webserver started at http://127.0.103.126:43757/ using document root <none> and password file <none>
I20250626 01:54:23.415109 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:23.415182 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:23.415321 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:23.416072 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/master-0-root/instance:
uuid: "08194e68dffb4b5299ee517ac4649748"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.418777 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20250626 01:54:23.420460 7065 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.420877 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20250626 01:54:23.420987 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/master-0-root
uuid: "08194e68dffb4b5299ee517ac4649748"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.421101 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:23.439088 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:23.439630 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:23.450789 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41555
I20250626 01:54:23.450812 7116 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41555 every 8 connection(s)
I20250626 01:54:23.452219 7117 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:23.456756 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Bootstrap starting.
I20250626 01:54:23.457969 7117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:23.459501 7117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: No bootstrap required, opened a new log
I20250626 01:54:23.460258 7117 raft_consensus.cc:357] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08194e68dffb4b5299ee517ac4649748" member_type: VOTER }
I20250626 01:54:23.460397 7117 raft_consensus.cc:383] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:23.460438 7117 raft_consensus.cc:738] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08194e68dffb4b5299ee517ac4649748, State: Initialized, Role: FOLLOWER
I20250626 01:54:23.460616 7117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08194e68dffb4b5299ee517ac4649748" member_type: VOTER }
I20250626 01:54:23.460731 7117 raft_consensus.cc:397] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250626 01:54:23.460772 7117 raft_consensus.cc:491] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250626 01:54:23.460834 7117 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:23.462634 7117 raft_consensus.cc:513] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08194e68dffb4b5299ee517ac4649748" member_type: VOTER }
I20250626 01:54:23.462818 7117 leader_election.cc:304] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08194e68dffb4b5299ee517ac4649748; no voters:
I20250626 01:54:23.463196 7117 leader_election.cc:290] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250626 01:54:23.463359 7120 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:23.463786 7120 raft_consensus.cc:695] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 1 LEADER]: Becoming Leader. State: Replica: 08194e68dffb4b5299ee517ac4649748, State: Running, Role: LEADER
I20250626 01:54:23.464046 7120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08194e68dffb4b5299ee517ac4649748" member_type: VOTER }
I20250626 01:54:23.464166 7117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [sys.catalog]: configured and running, proceeding with master startup.
I20250626 01:54:23.464921 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08194e68dffb4b5299ee517ac4649748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08194e68dffb4b5299ee517ac4649748" member_type: VOTER } }
I20250626 01:54:23.464980 7122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08194e68dffb4b5299ee517ac4649748. Latest consensus state: current_term: 1 leader_uuid: "08194e68dffb4b5299ee517ac4649748" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08194e68dffb4b5299ee517ac4649748" member_type: VOTER } }
I20250626 01:54:23.465193 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [sys.catalog]: This master's current role is: LEADER
I20250626 01:54:23.466485 7122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [sys.catalog]: This master's current role is: LEADER
I20250626 01:54:23.467281 7128 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250626 01:54:23.468463 7128 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250626 01:54:23.469592 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250626 01:54:23.471421 7128 catalog_manager.cc:1349] Generated new cluster ID: 6e974a85193043e396f9016914d6c69e
I20250626 01:54:23.471505 7128 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250626 01:54:23.486169 7128 catalog_manager.cc:1372] Generated new certificate authority record
I20250626 01:54:23.486994 7128 catalog_manager.cc:1506] Loading token signing keys...
I20250626 01:54:23.498099 7128 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Generated new TSK 0
I20250626 01:54:23.498363 7128 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250626 01:54:23.502525 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:23.505163 7138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.505666 7139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.505914 7141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:23.506017 413 server_base.cc:1048] running on GCE node
I20250626 01:54:23.506331 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:23.506390 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:23.506423 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902863506413 us; error 0 us; skew 500 ppm
I20250626 01:54:23.506561 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:23.507975 413 webserver.cc:469] Webserver started at http://127.0.103.65:41209/ using document root <none> and password file <none>
I20250626 01:54:23.508210 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:23.508278 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:23.508421 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:23.509110 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/instance:
uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.511839 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20250626 01:54:23.513615 7146 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.513989 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20250626 01:54:23.514076 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.514200 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:23.527092 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:23.527668 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:23.528303 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:23.529201 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:23.529256 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.529330 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:23.529377 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.541659 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:32973
I20250626 01:54:23.541680 7208 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:32973 every 8 connection(s)
I20250626 01:54:23.543417 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:23.546402 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.546736 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.547194 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:23.548170 413 server_base.cc:1048] running on GCE node
I20250626 01:54:23.548457 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:23.548520 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:23.548578 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902863548573 us; error 0 us; skew 500 ppm
I20250626 01:54:23.548725 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:23.550071 413 webserver.cc:469] Webserver started at http://127.0.103.66:36947/ using document root <none> and password file <none>
I20250626 01:54:23.550293 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:23.550384 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:23.550513 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:23.550531 7209 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41555
I20250626 01:54:23.550675 7209 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:23.551033 7209 heartbeater.cc:507] Master 127.0.103.126:41555 requested a full tablet report, sending...
I20250626 01:54:23.551185 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/instance:
uuid: "43874288b35e4f85adc362683201eb19"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.551873 7082 ts_manager.cc:194] Registered new tserver with Master: e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973)
I20250626 01:54:23.552969 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56796
I20250626 01:54:23.554009 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:54:23.555505 7221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.555824 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:23.555945 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root
uuid: "43874288b35e4f85adc362683201eb19"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.556056 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:23.569016 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:23.569605 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:23.570160 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:23.570966 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:23.571023 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.571102 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:23.571139 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.582996 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:37253
I20250626 01:54:23.583021 7283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:37253 every 8 connection(s)
I20250626 01:54:23.584625 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:23.587436 7288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.587955 7290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:23.588302 7287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:23.589109 413 server_base.cc:1048] running on GCE node
I20250626 01:54:23.589360 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:23.589423 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:23.589483 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902863589479 us; error 0 us; skew 500 ppm
I20250626 01:54:23.589646 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:23.591286 413 webserver.cc:469] Webserver started at http://127.0.103.67:36191/ using document root <none> and password file <none>
I20250626 01:54:23.591504 7284 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41555
I20250626 01:54:23.591534 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:23.591625 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:23.591636 7284 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:23.591759 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:23.592034 7284 heartbeater.cc:507] Master 127.0.103.126:41555 requested a full tablet report, sending...
I20250626 01:54:23.592602 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/instance:
uuid: "b3b91d8793f540749195348e46332124"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.592968 7082 ts_manager.cc:194] Registered new tserver with Master: 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:23.594075 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56810
I20250626 01:54:23.595600 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250626 01:54:23.597173 7295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.597503 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:23.597599 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root
uuid: "b3b91d8793f540749195348e46332124"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:23.597726 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:23.613169 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:23.613713 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:23.614279 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:23.615211 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:23.615271 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.615376 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:23.615418 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:23.627290 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:36473
I20250626 01:54:23.627302 7357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:36473 every 8 connection(s)
I20250626 01:54:23.633821 7358 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41555
I20250626 01:54:23.633971 7358 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:23.634321 7358 heartbeater.cc:507] Master 127.0.103.126:41555 requested a full tablet report, sending...
I20250626 01:54:23.635103 7082 ts_manager.cc:194] Registered new tserver with Master: b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:23.635177 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006866425s
I20250626 01:54:23.635872 7082 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:56816
I20250626 01:54:24.554494 7209 heartbeater.cc:499] Master 127.0.103.126:41555 was elected leader, sending a full tablet report...
I20250626 01:54:24.595544 7284 heartbeater.cc:499] Master 127.0.103.126:41555 was elected leader, sending a full tablet report...
I20250626 01:54:24.637210 7358 heartbeater.cc:499] Master 127.0.103.126:41555 was elected leader, sending a full tablet report...
I20250626 01:54:24.663455 413 test_util.cc:276] Using random seed: -683093097
I20250626 01:54:24.676354 7082 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56830:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:54:24.677155 7082 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:54:24.711230 7249 tablet_service.cc:1468] Processing CreateTablet for tablet fce1c61b02da4ff390f35254b24356c7 (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:24.711397 7248 tablet_service.cc:1468] Processing CreateTablet for tablet 26292b94e4ad4a6db496def22e462011 (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:24.711835 7249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce1c61b02da4ff390f35254b24356c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.711862 7323 tablet_service.cc:1468] Processing CreateTablet for tablet fce1c61b02da4ff390f35254b24356c7 (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:24.712342 7323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce1c61b02da4ff390f35254b24356c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.715600 7321 tablet_service.cc:1468] Processing CreateTablet for tablet 255285c177a147eabf811f0d749903dc (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:24.716156 7248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26292b94e4ad4a6db496def22e462011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.716156 7321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255285c177a147eabf811f0d749903dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.716262 7246 tablet_service.cc:1468] Processing CreateTablet for tablet 690913b2fc834a7bafecefcb9b823cab (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:24.716457 7320 tablet_service.cc:1468] Processing CreateTablet for tablet 690913b2fc834a7bafecefcb9b823cab (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:24.716667 7246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690913b2fc834a7bafecefcb9b823cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.716809 7320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690913b2fc834a7bafecefcb9b823cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.720402 7322 tablet_service.cc:1468] Processing CreateTablet for tablet 26292b94e4ad4a6db496def22e462011 (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:24.720903 7322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26292b94e4ad4a6db496def22e462011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.711351 7247 tablet_service.cc:1468] Processing CreateTablet for tablet 255285c177a147eabf811f0d749903dc (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:24.724525 7247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255285c177a147eabf811f0d749903dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.730741 7378 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:24.732790 7174 tablet_service.cc:1468] Processing CreateTablet for tablet fce1c61b02da4ff390f35254b24356c7 (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:24.733347 7174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce1c61b02da4ff390f35254b24356c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.734277 7378 tablet_bootstrap.cc:654] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.736092 7379 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:24.737613 7379 tablet_bootstrap.cc:654] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.738263 7171 tablet_service.cc:1468] Processing CreateTablet for tablet 690913b2fc834a7bafecefcb9b823cab (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:24.738705 7171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690913b2fc834a7bafecefcb9b823cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.739563 7378 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:24.739738 7382 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:24.739758 7378 ts_tablet_manager.cc:1397] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20250626 01:54:24.740432 7173 tablet_service.cc:1468] Processing CreateTablet for tablet 26292b94e4ad4a6db496def22e462011 (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:24.740860 7173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26292b94e4ad4a6db496def22e462011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.741523 7379 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:24.741860 7382 tablet_bootstrap.cc:654] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.741923 7379 ts_tablet_manager.cc:1397] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250626 01:54:24.742894 7379 raft_consensus.cc:357] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.743309 7379 raft_consensus.cc:383] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.743458 7379 raft_consensus.cc:738] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.740865 7378 raft_consensus.cc:357] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.743837 7378 raft_consensus.cc:383] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.743994 7378 raft_consensus.cc:738] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.744287 7378 consensus_queue.cc:260] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.744809 7379 consensus_queue.cc:260] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.746446 7378 ts_tablet_manager.cc:1428] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250626 01:54:24.746696 7172 tablet_service.cc:1468] Processing CreateTablet for tablet 255285c177a147eabf811f0d749903dc (DEFAULT_TABLE table=test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:24.746865 7382 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:24.746995 7382 ts_tablet_manager.cc:1397] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250626 01:54:24.747166 7172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255285c177a147eabf811f0d749903dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:24.747992 7378 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:24.749509 7382 raft_consensus.cc:357] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.749802 7378 tablet_bootstrap.cc:654] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.749783 7382 raft_consensus.cc:383] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.749967 7382 raft_consensus.cc:738] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.750331 7382 consensus_queue.cc:260] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.751370 7379 ts_tablet_manager.cc:1428] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250626 01:54:24.751621 7379 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:24.753773 7379 tablet_bootstrap.cc:654] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.755800 7378 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:24.756019 7378 ts_tablet_manager.cc:1397] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20250626 01:54:24.756937 7378 raft_consensus.cc:357] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.757148 7378 raft_consensus.cc:383] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.757289 7378 raft_consensus.cc:738] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.757570 7378 consensus_queue.cc:260] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.758139 7378 ts_tablet_manager.cc:1428] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:24.758466 7378 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:24.759042 7382 ts_tablet_manager.cc:1428] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20250626 01:54:24.759300 7382 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:24.760774 7382 tablet_bootstrap.cc:654] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.760880 7378 tablet_bootstrap.cc:654] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.769091 7379 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:24.769274 7379 ts_tablet_manager.cc:1397] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250626 01:54:24.769392 7378 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:24.769591 7378 ts_tablet_manager.cc:1397] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20250626 01:54:24.770074 7379 raft_consensus.cc:357] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.770241 7379 raft_consensus.cc:383] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.770303 7379 raft_consensus.cc:738] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.770553 7379 consensus_queue.cc:260] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.770705 7378 raft_consensus.cc:357] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.770941 7378 raft_consensus.cc:383] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.771021 7378 raft_consensus.cc:738] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.771026 7379 ts_tablet_manager.cc:1428] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:24.771317 7379 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:24.771286 7378 consensus_queue.cc:260] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.773030 7378 ts_tablet_manager.cc:1428] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250626 01:54:24.773430 7378 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:24.773556 7379 tablet_bootstrap.cc:654] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.776108 7378 tablet_bootstrap.cc:654] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.779556 7378 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:24.779556 7382 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:24.779780 7378 ts_tablet_manager.cc:1397] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20250626 01:54:24.779814 7382 ts_tablet_manager.cc:1397] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.008s
I20250626 01:54:24.780166 7379 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:24.780300 7379 ts_tablet_manager.cc:1397] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250626 01:54:24.780627 7382 raft_consensus.cc:357] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.780786 7382 raft_consensus.cc:383] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.780711 7378 raft_consensus.cc:357] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.780845 7382 raft_consensus.cc:738] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.780865 7378 raft_consensus.cc:383] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.780927 7378 raft_consensus.cc:738] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.780998 7379 raft_consensus.cc:357] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.781157 7379 raft_consensus.cc:383] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.781086 7382 consensus_queue.cc:260] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.781236 7379 raft_consensus.cc:738] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.781168 7378 consensus_queue.cc:260] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.781406 7379 consensus_queue.cc:260] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.781701 7378 ts_tablet_manager.cc:1428] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:24.782181 7382 ts_tablet_manager.cc:1428] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:24.782469 7382 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:24.783828 7379 ts_tablet_manager.cc:1428] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:54:24.784499 7382 tablet_bootstrap.cc:654] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.784627 7379 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:24.787242 7379 tablet_bootstrap.cc:654] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.789068 7379 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:24.789252 7379 ts_tablet_manager.cc:1397] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20250626 01:54:24.790158 7379 raft_consensus.cc:357] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.790331 7379 raft_consensus.cc:383] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.790387 7379 raft_consensus.cc:738] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.790591 7379 consensus_queue.cc:260] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.791504 7379 ts_tablet_manager.cc:1428] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:24.792500 7382 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:24.792654 7382 ts_tablet_manager.cc:1397] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20250626 01:54:24.793332 7382 raft_consensus.cc:357] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.793516 7382 raft_consensus.cc:383] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.793589 7382 raft_consensus.cc:738] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.793835 7382 consensus_queue.cc:260] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.794840 7382 ts_tablet_manager.cc:1428] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:24.795150 7382 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:24.797026 7382 tablet_bootstrap.cc:654] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:24.798982 7382 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:24.799104 7382 ts_tablet_manager.cc:1397] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250626 01:54:24.799718 7382 raft_consensus.cc:357] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.799865 7382 raft_consensus.cc:383] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:24.799973 7382 raft_consensus.cc:738] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:24.800164 7382 consensus_queue.cc:260] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.800565 7382 ts_tablet_manager.cc:1428] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:24.842152 7384 raft_consensus.cc:491] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:24.842361 7384 raft_consensus.cc:513] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.843156 7384 leader_election.cc:290] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:24.851475 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26292b94e4ad4a6db496def22e462011" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124" is_pre_election: true
I20250626 01:54:24.851475 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26292b94e4ad4a6db496def22e462011" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:24.851732 7184 raft_consensus.cc:2466] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:24.851729 7333 raft_consensus.cc:2466] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:24.852222 7222 leader_election.cc:304] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:24.852510 7384 raft_consensus.cc:2802] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:24.852596 7384 raft_consensus.cc:491] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:24.852682 7384 raft_consensus.cc:3058] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.854674 7384 raft_consensus.cc:513] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.855316 7384 leader_election.cc:290] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:24.855775 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26292b94e4ad4a6db496def22e462011" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:24.855803 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26292b94e4ad4a6db496def22e462011" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124"
I20250626 01:54:24.855998 7184 raft_consensus.cc:3058] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.856019 7333 raft_consensus.cc:3058] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.858189 7184 raft_consensus.cc:2466] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:24.858189 7333 raft_consensus.cc:2466] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:24.858659 7225 leader_election.cc:304] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:24.858959 7384 raft_consensus.cc:2802] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:24.859350 7384 raft_consensus.cc:695] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Becoming Leader. State: Replica: 43874288b35e4f85adc362683201eb19, State: Running, Role: LEADER
I20250626 01:54:24.859623 7384 consensus_queue.cc:237] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.862190 7082 catalog_manager.cc:5582] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43874288b35e4f85adc362683201eb19 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:24.872979 7385 raft_consensus.cc:491] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:24.873183 7385 raft_consensus.cc:513] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.873927 7385 leader_election.cc:290] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:24.880649 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "690913b2fc834a7bafecefcb9b823cab" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:24.880908 7184 raft_consensus.cc:2466] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:24.880992 7259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "690913b2fc834a7bafecefcb9b823cab" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19" is_pre_election: true
I20250626 01:54:24.881201 7259 raft_consensus.cc:2466] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:24.881261 7296 leader_election.cc:304] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:24.881503 7385 raft_consensus.cc:2802] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:24.881592 7385 raft_consensus.cc:491] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:24.881671 7385 raft_consensus.cc:3058] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.883595 7385 raft_consensus.cc:513] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.884176 7385 leader_election.cc:290] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:24.884709 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "690913b2fc834a7bafecefcb9b823cab" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:24.884742 7259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "690913b2fc834a7bafecefcb9b823cab" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19"
I20250626 01:54:24.884893 7259 raft_consensus.cc:3058] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.884893 7184 raft_consensus.cc:3058] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.887750 7184 raft_consensus.cc:2466] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:24.887750 7259 raft_consensus.cc:2466] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:24.888303 7296 leader_election.cc:304] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124; no voters:
I20250626 01:54:24.888597 7385 raft_consensus.cc:2802] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:24.889030 7385 raft_consensus.cc:695] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Becoming Leader. State: Replica: b3b91d8793f540749195348e46332124, State: Running, Role: LEADER
I20250626 01:54:24.889281 7385 consensus_queue.cc:237] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.892022 7082 catalog_manager.cc:5582] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b91d8793f540749195348e46332124 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } }
I20250626 01:54:24.911556 7384 raft_consensus.cc:491] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:24.911777 7384 raft_consensus.cc:513] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.912500 7384 leader_election.cc:290] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:24.913048 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "255285c177a147eabf811f0d749903dc" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124" is_pre_election: true
I20250626 01:54:24.913055 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "255285c177a147eabf811f0d749903dc" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:24.913247 7333 raft_consensus.cc:2466] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:24.913281 7184 raft_consensus.cc:2466] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:24.913650 7225 leader_election.cc:304] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:24.913933 7384 raft_consensus.cc:2802] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:24.914021 7384 raft_consensus.cc:491] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:24.914098 7384 raft_consensus.cc:3058] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.916286 7384 raft_consensus.cc:513] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.916940 7384 leader_election.cc:290] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:24.917330 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "255285c177a147eabf811f0d749903dc" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124"
I20250626 01:54:24.917315 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "255285c177a147eabf811f0d749903dc" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:24.917518 7184 raft_consensus.cc:3058] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.917516 7333 raft_consensus.cc:3058] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.920433 7333 raft_consensus.cc:2466] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:24.920433 7184 raft_consensus.cc:2466] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:24.920923 7222 leader_election.cc:304] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:24.921177 7384 raft_consensus.cc:2802] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:24.921299 7384 raft_consensus.cc:695] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Becoming Leader. State: Replica: 43874288b35e4f85adc362683201eb19, State: Running, Role: LEADER
I20250626 01:54:24.921559 7384 consensus_queue.cc:237] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:24.924345 7082 catalog_manager.cc:5582] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43874288b35e4f85adc362683201eb19 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:24.988564 7385 raft_consensus.cc:491] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:24.988808 7385 raft_consensus.cc:513] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.989439 7385 leader_election.cc:290] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253), e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973)
I20250626 01:54:24.990089 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fce1c61b02da4ff390f35254b24356c7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:24.990089 7259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fce1c61b02da4ff390f35254b24356c7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19" is_pre_election: true
I20250626 01:54:24.990322 7259 raft_consensus.cc:2466] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:24.990322 7184 raft_consensus.cc:2466] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:24.990722 7296 leader_election.cc:304] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124; no voters:
I20250626 01:54:24.990975 7385 raft_consensus.cc:2802] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:24.991058 7385 raft_consensus.cc:491] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:24.991127 7385 raft_consensus.cc:3058] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.993129 7385 raft_consensus.cc:513] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:24.993710 7385 leader_election.cc:290] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Requested vote from peers 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253), e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973)
I20250626 01:54:24.994272 7259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fce1c61b02da4ff390f35254b24356c7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19"
I20250626 01:54:24.994272 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fce1c61b02da4ff390f35254b24356c7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:24.994479 7259 raft_consensus.cc:3058] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.994479 7184 raft_consensus.cc:3058] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:24.997334 7259 raft_consensus.cc:2466] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:24.997334 7184 raft_consensus.cc:2466] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:24.997844 7296 leader_election.cc:304] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:24.998147 7385 raft_consensus.cc:2802] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:24.998279 7385 raft_consensus.cc:695] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Becoming Leader. State: Replica: b3b91d8793f540749195348e46332124, State: Running, Role: LEADER
I20250626 01:54:24.998536 7385 consensus_queue.cc:237] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:25.001310 7082 catalog_manager.cc:5582] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b91d8793f540749195348e46332124 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:25.020756 413 test_util.cc:276] Using random seed: -682735805
I20250626 01:54:25.033555 7082 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56840:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250626 01:54:25.034406 7082 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250626 01:54:25.043076 7172 tablet_service.cc:1468] Processing CreateTablet for tablet 60aec91189a247b786b0d20655c3843b (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:25.043140 7247 tablet_service.cc:1468] Processing CreateTablet for tablet 60aec91189a247b786b0d20655c3843b (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:25.043573 7247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60aec91189a247b786b0d20655c3843b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.044306 7321 tablet_service.cc:1468] Processing CreateTablet for tablet cba55e9c8b4f42c3a88586569eab312a (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:25.044699 7321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba55e9c8b4f42c3a88586569eab312a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.044812 7174 tablet_service.cc:1468] Processing CreateTablet for tablet c572611aa0b346dfa46a3035b5b4f7e7 (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:25.045413 7322 tablet_service.cc:1468] Processing CreateTablet for tablet 88b6214356474be7899829154958ec18 (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:25.045465 7323 tablet_service.cc:1468] Processing CreateTablet for tablet c572611aa0b346dfa46a3035b5b4f7e7 (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:25.045748 7322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b6214356474be7899829154958ec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.045886 7323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c572611aa0b346dfa46a3035b5b4f7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.051270 7378 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:25.051443 7171 tablet_service.cc:1468] Processing CreateTablet for tablet 88b6214356474be7899829154958ec18 (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:25.043159 7320 tablet_service.cc:1468] Processing CreateTablet for tablet 60aec91189a247b786b0d20655c3843b (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250626 01:54:25.052352 7320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60aec91189a247b786b0d20655c3843b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.052492 7379 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:25.053253 7378 tablet_bootstrap.cc:654] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.054158 7379 tablet_bootstrap.cc:654] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.054713 7173 tablet_service.cc:1468] Processing CreateTablet for tablet cba55e9c8b4f42c3a88586569eab312a (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:25.055397 7378 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:25.055554 7378 ts_tablet_manager.cc:1397] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250626 01:54:25.056229 7246 tablet_service.cc:1468] Processing CreateTablet for tablet 88b6214356474be7899829154958ec18 (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250626 01:54:25.056594 7246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b6214356474be7899829154958ec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.056558 7378 raft_consensus.cc:357] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.056783 7378 raft_consensus.cc:383] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.057086 7378 raft_consensus.cc:738] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.057358 7378 consensus_queue.cc:260] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.059537 7378 ts_tablet_manager.cc:1428] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250626 01:54:25.060434 7379 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:25.060506 7248 tablet_service.cc:1468] Processing CreateTablet for tablet c572611aa0b346dfa46a3035b5b4f7e7 (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250626 01:54:25.060858 7379 ts_tablet_manager.cc:1397] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20250626 01:54:25.060961 7248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c572611aa0b346dfa46a3035b5b4f7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.061834 7379 raft_consensus.cc:357] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.062114 7379 raft_consensus.cc:383] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.062266 7379 raft_consensus.cc:738] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.062631 7379 consensus_queue.cc:260] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.043579 7172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60aec91189a247b786b0d20655c3843b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.064396 7379 ts_tablet_manager.cc:1428] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250626 01:54:25.064851 7379 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:25.064433 7171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88b6214356474be7899829154958ec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.066917 7379 tablet_bootstrap.cc:654] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.067859 7174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c572611aa0b346dfa46a3035b5b4f7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.072052 7173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba55e9c8b4f42c3a88586569eab312a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.073493 7378 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:25.073781 7382 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:25.074122 7379 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:25.074362 7379 ts_tablet_manager.cc:1397] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20250626 01:54:25.075481 7382 tablet_bootstrap.cc:654] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.075359 7379 raft_consensus.cc:357] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.075477 7378 tablet_bootstrap.cc:654] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.078764 7379 raft_consensus.cc:383] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.078677 7247 tablet_service.cc:1468] Processing CreateTablet for tablet cba55e9c8b4f42c3a88586569eab312a (DEFAULT_TABLE table=dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250626 01:54:25.079110 7379 raft_consensus.cc:738] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.079372 7247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cba55e9c8b4f42c3a88586569eab312a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:25.079654 7379 consensus_queue.cc:260] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.082571 7379 ts_tablet_manager.cc:1428] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250626 01:54:25.083037 7379 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:25.085008 7379 tablet_bootstrap.cc:654] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.090173 7382 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:25.090391 7382 ts_tablet_manager.cc:1397] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20250626 01:54:25.090762 7378 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:25.090899 7378 ts_tablet_manager.cc:1397] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.011s
I20250626 01:54:25.091221 7382 raft_consensus.cc:357] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.091401 7382 raft_consensus.cc:383] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.091437 7378 raft_consensus.cc:357] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.091506 7382 raft_consensus.cc:738] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.091605 7378 raft_consensus.cc:383] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.091675 7378 raft_consensus.cc:738] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.091815 7382 consensus_queue.cc:260] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.091949 7378 consensus_queue.cc:260] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.092413 7378 ts_tablet_manager.cc:1428] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250626 01:54:25.093036 7378 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:25.093293 7379 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:25.093433 7379 ts_tablet_manager.cc:1397] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20250626 01:54:25.094326 7379 raft_consensus.cc:357] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.094516 7379 raft_consensus.cc:383] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.094617 7379 raft_consensus.cc:738] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.094892 7379 consensus_queue.cc:260] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.095423 7382 ts_tablet_manager.cc:1428] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250626 01:54:25.095535 7379 ts_tablet_manager.cc:1428] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250626 01:54:25.096036 7379 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Bootstrap starting.
I20250626 01:54:25.096822 7378 tablet_bootstrap.cc:654] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.097345 7382 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:25.099138 7382 tablet_bootstrap.cc:654] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.100099 7378 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:25.100282 7378 ts_tablet_manager.cc:1397] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20250626 01:54:25.101078 7379 tablet_bootstrap.cc:654] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.101154 7378 raft_consensus.cc:357] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.101372 7378 raft_consensus.cc:383] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.101473 7378 raft_consensus.cc:738] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.101716 7378 consensus_queue.cc:260] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.102267 7378 ts_tablet_manager.cc:1428] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:25.102571 7378 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Bootstrap starting.
I20250626 01:54:25.102576 7382 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:25.102749 7382 ts_tablet_manager.cc:1397] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250626 01:54:25.103561 7382 raft_consensus.cc:357] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.103788 7382 raft_consensus.cc:383] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.103916 7382 raft_consensus.cc:738] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.104180 7382 consensus_queue.cc:260] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.104758 7378 tablet_bootstrap.cc:654] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.104882 7382 ts_tablet_manager.cc:1428] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250626 01:54:25.105273 7382 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:25.105386 7379 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: No bootstrap required, opened a new log
I20250626 01:54:25.105521 7379 ts_tablet_manager.cc:1397] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250626 01:54:25.105844 7384 raft_consensus.cc:491] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:25.106016 7384 raft_consensus.cc:513] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.106389 7379 raft_consensus.cc:357] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.106592 7379 raft_consensus.cc:383] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.106668 7379 raft_consensus.cc:738] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b91d8793f540749195348e46332124, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.106906 7384 leader_election.cc:290] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:25.106956 7379 consensus_queue.cc:260] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.107199 7382 tablet_bootstrap.cc:654] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.107590 7379 ts_tablet_manager.cc:1428] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250626 01:54:25.108155 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60aec91189a247b786b0d20655c3843b" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:25.108417 7184 raft_consensus.cc:2466] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:25.109405 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60aec91189a247b786b0d20655c3843b" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124" is_pre_election: true
I20250626 01:54:25.109678 7333 raft_consensus.cc:2466] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:25.111611 7225 leader_election.cc:304] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124; no voters:
I20250626 01:54:25.113745 7384 raft_consensus.cc:2802] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:25.113932 7384 raft_consensus.cc:491] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:25.114063 7384 raft_consensus.cc:3058] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.117997 7384 raft_consensus.cc:513] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.118827 7378 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: No bootstrap required, opened a new log
I20250626 01:54:25.119019 7378 ts_tablet_manager.cc:1397] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.006s
I20250626 01:54:25.119438 7382 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:25.119385 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60aec91189a247b786b0d20655c3843b" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:25.119611 7184 raft_consensus.cc:3058] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.119601 7382 ts_tablet_manager.cc:1397] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.006s
I20250626 01:54:25.119594 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60aec91189a247b786b0d20655c3843b" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124"
I20250626 01:54:25.119865 7333 raft_consensus.cc:3058] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.120086 7378 raft_consensus.cc:357] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.120288 7378 raft_consensus.cc:383] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.120368 7378 raft_consensus.cc:738] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43874288b35e4f85adc362683201eb19, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.120509 7382 raft_consensus.cc:357] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.120680 7382 raft_consensus.cc:383] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.120633 7378 consensus_queue.cc:260] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.120783 7382 raft_consensus.cc:738] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.121008 7382 consensus_queue.cc:260] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.121376 7378 ts_tablet_manager.cc:1428] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250626 01:54:25.122541 7382 ts_tablet_manager.cc:1428] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250626 01:54:25.122865 7382 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:25.122869 7333 raft_consensus.cc:2466] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:25.123673 7384 leader_election.cc:290] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:25.123837 7225 leader_election.cc:304] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124; no voters:
I20250626 01:54:25.124702 7184 raft_consensus.cc:2466] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:25.124991 7382 tablet_bootstrap.cc:654] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Neither blocks nor log segments found. Creating new log.
I20250626 01:54:25.125743 7384 raft_consensus.cc:2802] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:25.126271 7384 raft_consensus.cc:695] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Becoming Leader. State: Replica: 43874288b35e4f85adc362683201eb19, State: Running, Role: LEADER
I20250626 01:54:25.126619 7384 consensus_queue.cc:237] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.127209 7382 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: No bootstrap required, opened a new log
I20250626 01:54:25.127413 7382 ts_tablet_manager.cc:1397] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250626 01:54:25.128402 7382 raft_consensus.cc:357] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.128630 7382 raft_consensus.cc:383] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250626 01:54:25.128739 7382 raft_consensus.cc:738] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:25.128985 7382 consensus_queue.cc:260] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.129415 7382 ts_tablet_manager.cc:1428] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:25.129644 7079 catalog_manager.cc:5582] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43874288b35e4f85adc362683201eb19 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:25.302037 7385 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.306447 7402 raft_consensus.cc:491] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:25.306574 7385 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.306687 7402 raft_consensus.cc:513] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.307368 7402 leader_election.cc:290] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:25.307626 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba55e9c8b4f42c3a88586569eab312a" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:25.308051 7184 raft_consensus.cc:2466] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:25.308629 7296 leader_election.cc:304] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:25.308733 7258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba55e9c8b4f42c3a88586569eab312a" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19" is_pre_election: true
I20250626 01:54:25.308918 7402 raft_consensus.cc:2802] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:25.308917 7258 raft_consensus.cc:2466] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:25.309064 7402 raft_consensus.cc:491] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:25.309170 7402 raft_consensus.cc:3058] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.311911 7402 raft_consensus.cc:513] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.312781 7402 leader_election.cc:290] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:25.313382 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba55e9c8b4f42c3a88586569eab312a" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:25.313454 7258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cba55e9c8b4f42c3a88586569eab312a" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19"
I20250626 01:54:25.313639 7184 raft_consensus.cc:3058] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.313635 7258 raft_consensus.cc:3058] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.316380 7258 raft_consensus.cc:2466] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:25.316807 7184 raft_consensus.cc:2466] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:25.316874 7296 leader_election.cc:304] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, b3b91d8793f540749195348e46332124; no voters:
I20250626 01:54:25.317209 7402 raft_consensus.cc:2802] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:25.317348 7402 raft_consensus.cc:695] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Becoming Leader. State: Replica: b3b91d8793f540749195348e46332124, State: Running, Role: LEADER
I20250626 01:54:25.317828 7402 consensus_queue.cc:237] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.320505 7079 catalog_manager.cc:5582] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b91d8793f540749195348e46332124 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } }
I20250626 01:54:25.322165 7384 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.326918 7398 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.333626 7398 raft_consensus.cc:491] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:25.333863 7398 raft_consensus.cc:513] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.334462 7398 leader_election.cc:290] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:25.334928 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88b6214356474be7899829154958ec18" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124" is_pre_election: true
I20250626 01:54:25.334941 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88b6214356474be7899829154958ec18" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:25.335127 7333 raft_consensus.cc:2466] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:25.335151 7184 raft_consensus.cc:2466] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 0.
I20250626 01:54:25.335510 7222 leader_election.cc:304] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:25.335813 7398 raft_consensus.cc:2802] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:25.335920 7398 raft_consensus.cc:491] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:25.336047 7398 raft_consensus.cc:3058] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.338212 7398 raft_consensus.cc:513] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.338824 7398 leader_election.cc:290] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:25.339313 7333 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88b6214356474be7899829154958ec18" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b91d8793f540749195348e46332124"
I20250626 01:54:25.339498 7333 raft_consensus.cc:3058] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.340111 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "88b6214356474be7899829154958ec18" candidate_uuid: "43874288b35e4f85adc362683201eb19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:25.340279 7184 raft_consensus.cc:3058] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.343068 7184 raft_consensus.cc:2466] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:25.343626 7222 leader_election.cc:304] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43874288b35e4f85adc362683201eb19, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:25.343994 7398 raft_consensus.cc:2802] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:25.344120 7398 raft_consensus.cc:695] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Becoming Leader. State: Replica: 43874288b35e4f85adc362683201eb19, State: Running, Role: LEADER
I20250626 01:54:25.344399 7398 consensus_queue.cc:237] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.347458 7080 catalog_manager.cc:5582] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 43874288b35e4f85adc362683201eb19 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:25.350878 7333 raft_consensus.cc:2466] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43874288b35e4f85adc362683201eb19 in term 1.
I20250626 01:54:25.396131 7398 consensus_queue.cc:1035] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.401317 7384 consensus_queue.cc:1035] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.444922 7402 raft_consensus.cc:491] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250626 01:54:25.445118 7402 raft_consensus.cc:513] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.445726 7402 leader_election.cc:290] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:25.446331 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c572611aa0b346dfa46a3035b5b4f7e7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" is_pre_election: true
I20250626 01:54:25.446358 7258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c572611aa0b346dfa46a3035b5b4f7e7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19" is_pre_election: true
I20250626 01:54:25.446502 7184 raft_consensus.cc:2466] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:25.446545 7258 raft_consensus.cc:2466] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 0.
I20250626 01:54:25.446903 7296 leader_election.cc:304] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:25.447136 7402 raft_consensus.cc:2802] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250626 01:54:25.447206 7402 raft_consensus.cc:491] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250626 01:54:25.447273 7402 raft_consensus.cc:3058] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.449288 7402 raft_consensus.cc:513] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.449853 7402 leader_election.cc:290] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Requested vote from peers e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973), 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:25.450423 7184 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c572611aa0b346dfa46a3035b5b4f7e7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
I20250626 01:54:25.450423 7258 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c572611aa0b346dfa46a3035b5b4f7e7" candidate_uuid: "b3b91d8793f540749195348e46332124" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43874288b35e4f85adc362683201eb19"
I20250626 01:54:25.450593 7184 raft_consensus.cc:3058] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.450657 7258 raft_consensus.cc:3058] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 0 FOLLOWER]: Advancing to term 1
I20250626 01:54:25.452639 7258 raft_consensus.cc:2466] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:25.452636 7184 raft_consensus.cc:2466] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b91d8793f540749195348e46332124 in term 1.
I20250626 01:54:25.453099 7296 leader_election.cc:304] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3b91d8793f540749195348e46332124, e5c18a4bcc584d15934a3c8ab2593fe9; no voters:
I20250626 01:54:25.453377 7402 raft_consensus.cc:2802] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Leader election won for term 1
I20250626 01:54:25.453526 7402 raft_consensus.cc:695] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Becoming Leader. State: Replica: b3b91d8793f540749195348e46332124, State: Running, Role: LEADER
I20250626 01:54:25.453799 7402 consensus_queue.cc:237] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:25.456409 7079 catalog_manager.cc:5582] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b91d8793f540749195348e46332124 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } }
I20250626 01:54:25.508596 7402 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.513347 7402 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.518848 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20250626 01:54:25.528847 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:25.529290 413 tablet_replica.cc:331] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.529613 413 raft_consensus.cc:2241] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.529811 413 raft_consensus.cc:2270] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.530583 413 tablet_replica.cc:331] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.530723 413 raft_consensus.cc:2241] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.530848 413 raft_consensus.cc:2270] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.531383 413 tablet_replica.cc:331] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.531509 413 raft_consensus.cc:2241] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.531636 413 raft_consensus.cc:2270] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.532337 413 tablet_replica.cc:331] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.532470 413 raft_consensus.cc:2241] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.532598 413 raft_consensus.cc:2270] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.533170 413 tablet_replica.cc:331] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.533284 413 raft_consensus.cc:2241] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.533386 413 raft_consensus.cc:2270] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.533859 413 tablet_replica.cc:331] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.533967 413 raft_consensus.cc:2241] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.534076 413 raft_consensus.cc:2270] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.534821 413 tablet_replica.cc:331] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.534932 413 raft_consensus.cc:2241] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.535043 413 raft_consensus.cc:2270] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.535614 413 tablet_replica.cc:331] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: stopping tablet replica
I20250626 01:54:25.535722 413 raft_consensus.cc:2241] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:25.535838 413 raft_consensus.cc:2270] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:25.552170 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
W20250626 01:54:25.654022 7222 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111) [suppressed 14 similar messages]
W20250626 01:54:25.655198 7222 consensus_peers.cc:487] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:25.684295 7384 consensus_queue.cc:1035] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:25.755273 7296 consensus_peers.cc:487] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 01:54:25.773947 7296 consensus_peers.cc:487] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 01:54:25.822537 7222 consensus_peers.cc:487] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 01:54:25.887301 7222 consensus_peers.cc:487] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:25.908108 7439 consensus_queue.cc:1035] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250626 01:54:25.908963 7384 consensus_queue.cc:1035] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:25.944222 7222 consensus_peers.cc:487] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:25.971458 7402 consensus_queue.cc:1035] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250626 01:54:25.976722 7296 consensus_peers.cc:487] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250626 01:54:25.998775 7296 consensus_peers.cc:487] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.337075 7385 consensus_queue.cc:579] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Leader has been unable to successfully communicate with peer e5c18a4bcc584d15934a3c8ab2593fe9 for more than 1 seconds (1.030s)
W20250626 01:54:26.339977 7079 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 690913b2fc834a7bafecefcb9b823cab with cas_config_opid_index -1: no extra replica candidate found for tablet 690913b2fc834a7bafecefcb9b823cab (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250626 01:54:26.393440 7384 consensus_queue.cc:579] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [LEADER]: Leader has been unable to successfully communicate with peer e5c18a4bcc584d15934a3c8ab2593fe9 for more than 1 seconds (1.049s)
I20250626 01:54:26.490137 7385 consensus_queue.cc:579] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [LEADER]: Leader has been unable to successfully communicate with peer e5c18a4bcc584d15934a3c8ab2593fe9 for more than 1 seconds (1.036s)
I20250626 01:54:26.565081 7384 consensus_queue.cc:579] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Leader has been unable to successfully communicate with peer e5c18a4bcc584d15934a3c8ab2593fe9 for more than 1 seconds (1.161s)
I20250626 01:54:26.565562 7079 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56830:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250626 01:54:26.565784 7079 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56830:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250626 01:54:26.571573 7079 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Sending DeleteTablet for 3 replicas of tablet 60aec91189a247b786b0d20655c3843b
I20250626 01:54:26.572244 7079 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Sending DeleteTablet for 3 replicas of tablet cba55e9c8b4f42c3a88586569eab312a
I20250626 01:54:26.572779 7079 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Sending DeleteTablet for 3 replicas of tablet 88b6214356474be7899829154958ec18
I20250626 01:54:26.572815 7320 tablet_service.cc:1515] Processing DeleteTablet for tablet 60aec91189a247b786b0d20655c3843b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:51580
I20250626 01:54:26.573128 7321 tablet_service.cc:1515] Processing DeleteTablet for tablet cba55e9c8b4f42c3a88586569eab312a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:51580
I20250626 01:54:26.573215 7079 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: Sending DeleteTablet for 3 replicas of tablet c572611aa0b346dfa46a3035b5b4f7e7
I20250626 01:54:26.574427 7247 tablet_service.cc:1515] Processing DeleteTablet for tablet 60aec91189a247b786b0d20655c3843b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46408
I20250626 01:54:26.574834 7322 tablet_service.cc:1515] Processing DeleteTablet for tablet c572611aa0b346dfa46a3035b5b4f7e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:51580
I20250626 01:54:26.574939 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250626 01:54:26.574844 7456 tablet_replica.cc:331] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:26.575177 7457 tablet_replica.cc:331] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:26.575263 7246 tablet_service.cc:1515] Processing DeleteTablet for tablet 88b6214356474be7899829154958ec18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46408
I20250626 01:54:26.575235 7456 raft_consensus.cc:2241] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.575503 7248 tablet_service.cc:1515] Processing DeleteTablet for tablet cba55e9c8b4f42c3a88586569eab312a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46408
I20250626 01:54:26.575196 7249 tablet_service.cc:1515] Processing DeleteTablet for tablet c572611aa0b346dfa46a3035b5b4f7e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46408
I20250626 01:54:26.575727 7321 tablet_service.cc:1515] Processing DeleteTablet for tablet 88b6214356474be7899829154958ec18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:51580
I20250626 01:54:26.575608 7456 raft_consensus.cc:2270] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.575572 7457 raft_consensus.cc:2241] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Raft consensus shutting down.
W20250626 01:54:26.576804 7066 catalog_manager.cc:4671] TS e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c572611aa0b346dfa46a3035b5b4f7e7: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111)
I20250626 01:54:26.577150 7456 ts_tablet_manager.cc:1905] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.576788 7457 raft_consensus.cc:2270] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.578589 7457 ts_tablet_manager.cc:1905] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250626 01:54:26.578711 7459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:26.578950 7460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:26.587131 413 server_base.cc:1048] running on GCE node
W20250626 01:54:26.587236 7462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:26.587647 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:26.587738 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:26.587783 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902866587779 us; error 0 us; skew 500 ppm
I20250626 01:54:26.587977 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:26.588107 7456 ts_tablet_manager.cc:1918] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.588551 7456 log.cc:1199] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/wals/60aec91189a247b786b0d20655c3843b
I20250626 01:54:26.589082 7456 ts_tablet_manager.cc:1939] T 60aec91189a247b786b0d20655c3843b P b3b91d8793f540749195348e46332124: Deleting consensus metadata
I20250626 01:54:26.589758 7457 ts_tablet_manager.cc:1918] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.589869 7457 log.cc:1199] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/wals/60aec91189a247b786b0d20655c3843b
I20250626 01:54:26.590232 7456 tablet_replica.cc:331] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:26.590330 7457 ts_tablet_manager.cc:1939] T 60aec91189a247b786b0d20655c3843b P 43874288b35e4f85adc362683201eb19: Deleting consensus metadata
I20250626 01:54:26.590363 7069 catalog_manager.cc:4928] TS b3b91d8793f540749195348e46332124 (127.0.103.67:36473): tablet 60aec91189a247b786b0d20655c3843b (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.590425 7456 raft_consensus.cc:2241] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:26.590756 7456 raft_consensus.cc:2270] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.591485 7456 ts_tablet_manager.cc:1905] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.591761 7066 catalog_manager.cc:4928] TS 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253): tablet 60aec91189a247b786b0d20655c3843b (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.592160 7457 tablet_replica.cc:331] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:26.592142 413 webserver.cc:469] Webserver started at http://127.0.103.68:36517/ using document root <none> and password file <none>
I20250626 01:54:26.592384 7457 raft_consensus.cc:2241] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:26.592463 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:26.592530 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250626 01:54:26.592671 413 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250626 01:54:26.592730 7457 raft_consensus.cc:2270] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.593510 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root/instance:
uuid: "882d449e16bd4578ab034cc8c7e0e3c4"
format_stamp: "Formatted at 2025-06-26 01:54:26 on dist-test-slave-psm5"
I20250626 01:54:26.593746 7457 ts_tablet_manager.cc:1905] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.596733 7456 ts_tablet_manager.cc:1918] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.596923 7456 log.cc:1199] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/wals/cba55e9c8b4f42c3a88586569eab312a
I20250626 01:54:26.597178 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250626 01:54:26.597312 7456 ts_tablet_manager.cc:1939] T cba55e9c8b4f42c3a88586569eab312a P b3b91d8793f540749195348e46332124: Deleting consensus metadata
I20250626 01:54:26.598239 7456 tablet_replica.cc:331] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:26.598430 7456 raft_consensus.cc:2241] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:26.598443 7069 catalog_manager.cc:4928] TS b3b91d8793f540749195348e46332124 (127.0.103.67:36473): tablet cba55e9c8b4f42c3a88586569eab312a (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.598768 7456 raft_consensus.cc:2270] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.599591 7456 ts_tablet_manager.cc:1905] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.600378 7467 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:26.600853 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:26.600967 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root
uuid: "882d449e16bd4578ab034cc8c7e0e3c4"
format_stamp: "Formatted at 2025-06-26 01:54:26 on dist-test-slave-psm5"
I20250626 01:54:26.601090 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:26.601514 7457 ts_tablet_manager.cc:1918] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.601608 7457 log.cc:1199] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/wals/88b6214356474be7899829154958ec18
I20250626 01:54:26.602053 7457 ts_tablet_manager.cc:1939] T 88b6214356474be7899829154958ec18 P 43874288b35e4f85adc362683201eb19: Deleting consensus metadata
I20250626 01:54:26.603148 7066 catalog_manager.cc:4928] TS 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253): tablet 88b6214356474be7899829154958ec18 (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.603256 7457 tablet_replica.cc:331] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:26.603534 7457 raft_consensus.cc:2241] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.603760 7457 raft_consensus.cc:2270] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.604895 7457 ts_tablet_manager.cc:1905] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.605576 7456 ts_tablet_manager.cc:1918] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.605672 7456 log.cc:1199] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/wals/c572611aa0b346dfa46a3035b5b4f7e7
I20250626 01:54:26.606015 7456 ts_tablet_manager.cc:1939] T c572611aa0b346dfa46a3035b5b4f7e7 P b3b91d8793f540749195348e46332124: Deleting consensus metadata
I20250626 01:54:26.606824 7456 tablet_replica.cc:331] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:26.606997 7456 raft_consensus.cc:2241] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.606994 7069 catalog_manager.cc:4928] TS b3b91d8793f540749195348e46332124 (127.0.103.67:36473): tablet c572611aa0b346dfa46a3035b5b4f7e7 (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.607169 7456 raft_consensus.cc:2270] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.608292 7456 ts_tablet_manager.cc:1905] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.612512 7457 ts_tablet_manager.cc:1918] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.612643 7457 log.cc:1199] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/wals/cba55e9c8b4f42c3a88586569eab312a
I20250626 01:54:26.613030 7457 ts_tablet_manager.cc:1939] T cba55e9c8b4f42c3a88586569eab312a P 43874288b35e4f85adc362683201eb19: Deleting consensus metadata
I20250626 01:54:26.613341 7456 ts_tablet_manager.cc:1918] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.613471 7456 log.cc:1199] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/wals/88b6214356474be7899829154958ec18
I20250626 01:54:26.613791 7456 ts_tablet_manager.cc:1939] T 88b6214356474be7899829154958ec18 P b3b91d8793f540749195348e46332124: Deleting consensus metadata
I20250626 01:54:26.614146 7066 catalog_manager.cc:4928] TS 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253): tablet cba55e9c8b4f42c3a88586569eab312a (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.614213 7457 tablet_replica.cc:331] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:26.614384 7457 raft_consensus.cc:2241] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.614537 7457 raft_consensus.cc:2270] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.614823 7069 catalog_manager.cc:4928] TS b3b91d8793f540749195348e46332124 (127.0.103.67:36473): tablet 88b6214356474be7899829154958ec18 (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.615489 7457 ts_tablet_manager.cc:1905] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.620393 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:26.621104 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:26.621799 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:26.622896 413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250626 01:54:26.622974 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:26.623050 413 ts_tablet_manager.cc:610] Registered 0 tablets
I20250626 01:54:26.623103 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:26.637586 7457 ts_tablet_manager.cc:1918] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250626 01:54:26.637748 7457 log.cc:1199] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/wals/c572611aa0b346dfa46a3035b5b4f7e7
I20250626 01:54:26.638098 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:36707
I20250626 01:54:26.638108 7529 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:36707 every 8 connection(s)
I20250626 01:54:26.638206 7457 ts_tablet_manager.cc:1939] T c572611aa0b346dfa46a3035b5b4f7e7 P 43874288b35e4f85adc362683201eb19: Deleting consensus metadata
I20250626 01:54:26.639454 7066 catalog_manager.cc:4928] TS 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253): tablet c572611aa0b346dfa46a3035b5b4f7e7 (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.639775 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250626 01:54:26.642556 7533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:26.643322 7536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250626 01:54:26.643513 7534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250626 01:54:26.644136 413 server_base.cc:1048] running on GCE node
I20250626 01:54:26.644428 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250626 01:54:26.644495 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250626 01:54:26.644542 413 hybrid_clock.cc:648] HybridClock initialized: now 1750902866644538 us; error 0 us; skew 500 ppm
I20250626 01:54:26.644740 413 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250626 01:54:26.646131 413 webserver.cc:469] Webserver started at http://127.0.103.65:41209/ using document root <none> and password file <none>
I20250626 01:54:26.646404 413 fs_manager.cc:362] Metadata directory not provided
I20250626 01:54:26.646500 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250626 01:54:26.646566 7530 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41555
I20250626 01:54:26.646698 7530 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:26.647028 7530 heartbeater.cc:507] Master 127.0.103.126:41555 requested a full tablet report, sending...
I20250626 01:54:26.647936 7079 ts_manager.cc:194] Registered new tserver with Master: 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707)
I20250626 01:54:26.648766 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:26.648998 7079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48832
I20250626 01:54:26.650818 7541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250626 01:54:26.651156 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20250626 01:54:26.651252 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root
uuid: "e5c18a4bcc584d15934a3c8ab2593fe9"
format_stamp: "Formatted at 2025-06-26 01:54:23 on dist-test-slave-psm5"
I20250626 01:54:26.651367 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250626 01:54:26.656742 7258 consensus_queue.cc:237] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:26.659395 7333 raft_consensus.cc:1273] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:54:26.660162 7398 consensus_queue.cc:1035] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250626 01:54:26.661345 7222 consensus_peers.cc:487] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.663264 7384 raft_consensus.cc:2953] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.664515 7333 raft_consensus.cc:2953] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
W20250626 01:54:26.667577 7222 consensus_peers.cc:487] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26292b94e4ad4a6db496def22e462011. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.667991 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250626 01:54:26.668761 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250626 01:54:26.668823 7066 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 26292b94e4ad4a6db496def22e462011 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 3)
I20250626 01:54:26.669521 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250626 01:54:26.669945 7079 catalog_manager.cc:5582] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 reported cstate change: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.670989 7552 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20250626 01:54:26.688601 413 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250626 01:54:26.688724 413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20250626 01:54:26.688819 413 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250626 01:54:26.690862 7552 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.695257 7552 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.695647 7552 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.695871 7552 ts_tablet_manager.cc:1397] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250626 01:54:26.696585 7552 raft_consensus.cc:357] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.696728 7552 raft_consensus.cc:738] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.696939 7552 consensus_queue.cc:260] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.697630 7552 ts_tablet_manager.cc:1428] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:54:26.697846 7552 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.701754 7552 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.702104 7552 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.702314 7552 ts_tablet_manager.cc:1397] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250626 01:54:26.703027 413 ts_tablet_manager.cc:610] Registered 8 tablets
I20250626 01:54:26.702962 7552 raft_consensus.cc:357] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.703094 413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.014s user 0.011s sys 0.000s
I20250626 01:54:26.703114 7552 raft_consensus.cc:738] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.703316 7552 consensus_queue.cc:260] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.703703 7552 ts_tablet_manager.cc:1428] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:54:26.703954 7552 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.707751 7552 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.708201 7552 tablet_bootstrap.cc:492] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.708499 7552 ts_tablet_manager.cc:1397] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250626 01:54:26.709364 7552 raft_consensus.cc:357] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.709512 7552 raft_consensus.cc:738] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.709738 7552 consensus_queue.cc:260] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.709877 7333 consensus_queue.cc:237] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:26.710213 7552 ts_tablet_manager.cc:1428] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:26.710438 7552 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.715710 7258 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:54:26.717005 7402 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:54:26.719574 7552 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.720086 7552 tablet_bootstrap.cc:492] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
W20250626 01:54:26.720045 7296 consensus_peers.cc:487] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.720678 7552 ts_tablet_manager.cc:1397] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20250626 01:54:26.721457 7552 raft_consensus.cc:357] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.721732 7552 raft_consensus.cc:738] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.722029 7552 consensus_queue.cc:260] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
W20250626 01:54:26.722370 7296 consensus_peers.cc:487] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 690913b2fc834a7bafecefcb9b823cab. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.722898 7552 ts_tablet_manager.cc:1428] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250626 01:54:26.723131 7552 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.724498 7402 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.725054 7258 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.728036 7552 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.728570 7552 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.729516 7552 ts_tablet_manager.cc:1397] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250626 01:54:26.730206 7552 raft_consensus.cc:357] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:26.730449 7552 raft_consensus.cc:738] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.730763 7552 consensus_queue.cc:260] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:26.730736 7080 catalog_manager.cc:5582] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.731351 7552 ts_tablet_manager.cc:1428] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:26.731597 7552 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.734016 7069 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 690913b2fc834a7bafecefcb9b823cab with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250626 01:54:26.736907 7552 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.737212 7552 tablet_bootstrap.cc:492] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.737429 7552 ts_tablet_manager.cc:1397] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20250626 01:54:26.738006 7552 raft_consensus.cc:357] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.738162 7552 raft_consensus.cc:738] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.738420 7552 consensus_queue.cc:260] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.738947 7552 ts_tablet_manager.cc:1428] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20250626 01:54:26.739214 7552 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.742993 7552 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.743345 7552 tablet_bootstrap.cc:492] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.743618 7552 ts_tablet_manager.cc:1397] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250626 01:54:26.744261 7435 consensus_queue.cc:579] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Leader has been unable to successfully communicate with peer e5c18a4bcc584d15934a3c8ab2593fe9 for more than 1 seconds (1.414s)
I20250626 01:54:26.744416 7552 raft_consensus.cc:357] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.744746 7552 raft_consensus.cc:738] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.745024 7552 consensus_queue.cc:260] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.745651 7552 ts_tablet_manager.cc:1428] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:26.745936 7552 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap starting.
I20250626 01:54:26.748939 7258 consensus_queue.cc:237] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:26.751688 7552 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:26.752270 7552 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Bootstrap complete.
I20250626 01:54:26.752591 7552 ts_tablet_manager.cc:1397] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20250626 01:54:26.752743 7333 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 01:54:26.752815 7222 consensus_peers.cc:487] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 255285c177a147eabf811f0d749903dc. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.753444 7552 raft_consensus.cc:357] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:26.753525 7398 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:54:26.753686 7552 raft_consensus.cc:738] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5c18a4bcc584d15934a3c8ab2593fe9, State: Initialized, Role: FOLLOWER
I20250626 01:54:26.753942 7552 consensus_queue.cc:260] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
W20250626 01:54:26.754097 7222 consensus_peers.cc:487] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 -> Peer e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): Couldn't send request to peer e5c18a4bcc584d15934a3c8ab2593fe9. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:32973: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.754684 7552 ts_tablet_manager.cc:1428] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250626 01:54:26.756899 7384 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.758488 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:32973
I20250626 01:54:26.758517 7622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:32973 every 8 connection(s)
I20250626 01:54:26.761706 7066 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 255285c177a147eabf811f0d749903dc with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:54:26.761876 7333 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.763200 7080 catalog_manager.cc:5582] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:26.767055 7623 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41555
I20250626 01:54:26.767261 7623 heartbeater.cc:461] Registering TS with master...
I20250626 01:54:26.767731 7623 heartbeater.cc:507] Master 127.0.103.126:41555 requested a full tablet report, sending...
I20250626 01:54:26.769423 7079 ts_manager.cc:194] Re-registered known tserver with Master: e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973)
I20250626 01:54:26.772258 7079 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48842
I20250626 01:54:26.776290 7581 tablet_service.cc:1515] Processing DeleteTablet for tablet c572611aa0b346dfa46a3035b5b4f7e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46694
I20250626 01:54:26.776894 7627 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:26.777101 7627 raft_consensus.cc:2241] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.777290 7627 raft_consensus.cc:2270] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.777971 7627 ts_tablet_manager.cc:1905] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.782994 7627 ts_tablet_manager.cc:1918] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 01:54:26.783111 7627 log.cc:1199] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/wals/c572611aa0b346dfa46a3035b5b4f7e7
I20250626 01:54:26.783388 7627 ts_tablet_manager.cc:1939] T c572611aa0b346dfa46a3035b5b4f7e7 P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting consensus metadata
I20250626 01:54:26.784515 7066 catalog_manager.cc:4928] TS e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): tablet c572611aa0b346dfa46a3035b5b4f7e7 (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.793630 7581 tablet_service.cc:1515] Processing DeleteTablet for tablet 88b6214356474be7899829154958ec18 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46694
I20250626 01:54:26.794030 7627 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:26.794226 7627 raft_consensus.cc:2241] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.794392 7627 raft_consensus.cc:2270] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.795199 7627 ts_tablet_manager.cc:1905] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.800006 7627 ts_tablet_manager.cc:1918] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 01:54:26.800135 7627 log.cc:1199] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/wals/88b6214356474be7899829154958ec18
I20250626 01:54:26.800417 7627 ts_tablet_manager.cc:1939] T 88b6214356474be7899829154958ec18 P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting consensus metadata
I20250626 01:54:26.801415 7066 catalog_manager.cc:4928] TS e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): tablet 88b6214356474be7899829154958ec18 (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.807649 7581 tablet_service.cc:1515] Processing DeleteTablet for tablet 60aec91189a247b786b0d20655c3843b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46694
I20250626 01:54:26.808072 7627 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:26.808251 7627 raft_consensus.cc:2241] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.808421 7627 raft_consensus.cc:2270] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.809224 7627 ts_tablet_manager.cc:1905] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.814199 7627 ts_tablet_manager.cc:1918] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 01:54:26.814327 7627 log.cc:1199] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/wals/60aec91189a247b786b0d20655c3843b
I20250626 01:54:26.814548 7627 ts_tablet_manager.cc:1939] T 60aec91189a247b786b0d20655c3843b P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting consensus metadata
I20250626 01:54:26.815588 7066 catalog_manager.cc:4928] TS e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): tablet 60aec91189a247b786b0d20655c3843b (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.872252 7581 tablet_service.cc:1515] Processing DeleteTablet for tablet cba55e9c8b4f42c3a88586569eab312a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-26 01:54:26 UTC) from {username='slave'} at 127.0.0.1:46694
I20250626 01:54:26.872625 7627 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:26.872785 7627 raft_consensus.cc:2241] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:26.872949 7627 raft_consensus.cc:2270] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:26.873657 7627 ts_tablet_manager.cc:1905] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250626 01:54:26.878175 7627 ts_tablet_manager.cc:1918] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250626 01:54:26.878291 7627 log.cc:1199] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/wals/cba55e9c8b4f42c3a88586569eab312a
I20250626 01:54:26.878522 7627 ts_tablet_manager.cc:1939] T cba55e9c8b4f42c3a88586569eab312a P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting consensus metadata
I20250626 01:54:26.879601 7066 catalog_manager.cc:4928] TS e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): tablet cba55e9c8b4f42c3a88586569eab312a (table dugtrio [id=3ec4ed9748c74e56820def833f2bf2cf]) successfully deleted
I20250626 01:54:26.970985 7402 consensus_queue.cc:579] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Leader has been unable to successfully communicate with peer e5c18a4bcc584d15934a3c8ab2593fe9 for more than 1 seconds (1.455s)
I20250626 01:54:26.974563 7333 consensus_queue.cc:237] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:26.976918 7258 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250626 01:54:26.977097 7296 consensus_peers.cc:487] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fce1c61b02da4ff390f35254b24356c7. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:26.977275 7594 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250626 01:54:26.977420 7623 heartbeater.cc:499] Master 127.0.103.126:41555 was elected leader, sending a full tablet report...
I20250626 01:54:26.977723 7402 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:54:26.977994 7385 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250626 01:54:26.981269 7385 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.981992 7594 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.982517 7258 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.985713 7069 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fce1c61b02da4ff390f35254b24356c7 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250626 01:54:26.986387 7079 catalog_manager.cc:5582] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 reported cstate change: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:26.990809 7333 consensus_queue.cc:237] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } }
I20250626 01:54:26.992444 7594 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:26.992653 7258 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:26.993130 7439 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:26.993335 7385 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:26.995656 7385 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } }
I20250626 01:54:26.996222 7258 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } }
I20250626 01:54:26.996254 7594 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } }
I20250626 01:54:27.000856 7081 catalog_manager.cc:5582] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } }
I20250626 01:54:27.002147 7069 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fce1c61b02da4ff390f35254b24356c7 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:27.009572 7495 tablet_service.cc:1515] Processing DeleteTablet for tablet fce1c61b02da4ff390f35254b24356c7 with delete_type TABLET_DATA_TOMBSTONED (TS 882d449e16bd4578ab034cc8c7e0e3c4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43632
W20250626 01:54:27.010661 7066 catalog_manager.cc:4895] TS 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): delete failed for tablet fce1c61b02da4ff390f35254b24356c7 because the tablet was not found. No further retry: Not found: Tablet not found: fce1c61b02da4ff390f35254b24356c7
I20250626 01:54:27.043182 7594 raft_consensus.cc:2953] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:27.144758 7634 ts_tablet_manager.cc:927] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Initiating tablet copy from peer 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:27.145310 7634 tablet_copy_client.cc:323] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:37253
I20250626 01:54:27.150470 7269 tablet_copy_service.cc:140] P 43874288b35e4f85adc362683201eb19: Received BeginTabletCopySession request for tablet 26292b94e4ad4a6db496def22e462011 from peer 882d449e16bd4578ab034cc8c7e0e3c4 ({username='slave'} at 127.0.0.1:33708)
I20250626 01:54:27.150673 7269 tablet_copy_service.cc:161] P 43874288b35e4f85adc362683201eb19: Beginning new tablet copy session on tablet 26292b94e4ad4a6db496def22e462011 from peer 882d449e16bd4578ab034cc8c7e0e3c4 at {username='slave'} at 127.0.0.1:33708: session id = 882d449e16bd4578ab034cc8c7e0e3c4-26292b94e4ad4a6db496def22e462011
I20250626 01:54:27.152618 7269 tablet_copy_source_session.cc:215] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:27.153905 7634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26292b94e4ad4a6db496def22e462011. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:27.158622 7634 tablet_copy_client.cc:806] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:27.158839 7634 tablet_copy_client.cc:670] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:27.160408 7634 tablet_copy_client.cc:538] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:27.162966 7634 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap starting.
I20250626 01:54:27.168056 7634 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:27.168442 7634 tablet_bootstrap.cc:492] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap complete.
I20250626 01:54:27.168690 7634 ts_tablet_manager.cc:1397] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20250626 01:54:27.169364 7634 raft_consensus.cc:357] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:27.169541 7634 raft_consensus.cc:738] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 882d449e16bd4578ab034cc8c7e0e3c4, State: Initialized, Role: LEARNER
I20250626 01:54:27.169741 7634 consensus_queue.cc:260] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:27.170421 7634 ts_tablet_manager.cc:1428] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250626 01:54:27.170487 7530 heartbeater.cc:499] Master 127.0.103.126:41555 was elected leader, sending a full tablet report...
I20250626 01:54:27.171252 7269 tablet_copy_service.cc:342] P 43874288b35e4f85adc362683201eb19: Request end of tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-26292b94e4ad4a6db496def22e462011 received from {username='slave'} at 127.0.0.1:33708
I20250626 01:54:27.171391 7269 tablet_copy_service.cc:434] P 43874288b35e4f85adc362683201eb19: ending tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-26292b94e4ad4a6db496def22e462011 on tablet 26292b94e4ad4a6db496def22e462011 with peer 882d449e16bd4578ab034cc8c7e0e3c4
I20250626 01:54:27.218106 7594 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:27.227825 7594 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:27.294709 7634 ts_tablet_manager.cc:927] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Initiating tablet copy from peer b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:27.295399 7634 tablet_copy_client.cc:323] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:36473
I20250626 01:54:27.302579 7343 tablet_copy_service.cc:140] P b3b91d8793f540749195348e46332124: Received BeginTabletCopySession request for tablet 690913b2fc834a7bafecefcb9b823cab from peer 882d449e16bd4578ab034cc8c7e0e3c4 ({username='slave'} at 127.0.0.1:57854)
I20250626 01:54:27.302767 7343 tablet_copy_service.cc:161] P b3b91d8793f540749195348e46332124: Beginning new tablet copy session on tablet 690913b2fc834a7bafecefcb9b823cab from peer 882d449e16bd4578ab034cc8c7e0e3c4 at {username='slave'} at 127.0.0.1:57854: session id = 882d449e16bd4578ab034cc8c7e0e3c4-690913b2fc834a7bafecefcb9b823cab
I20250626 01:54:27.304973 7343 tablet_copy_source_session.cc:215] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:27.306283 7634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690913b2fc834a7bafecefcb9b823cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:27.310879 7634 tablet_copy_client.cc:806] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:27.311108 7634 tablet_copy_client.cc:670] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:27.312702 7634 tablet_copy_client.cc:538] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:27.315270 7634 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap starting.
I20250626 01:54:27.320315 7634 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:27.320657 7634 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap complete.
I20250626 01:54:27.320863 7634 ts_tablet_manager.cc:1397] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20250626 01:54:27.321467 7634 raft_consensus.cc:357] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:27.321625 7634 raft_consensus.cc:738] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 882d449e16bd4578ab034cc8c7e0e3c4, State: Initialized, Role: LEARNER
I20250626 01:54:27.321758 7634 consensus_queue.cc:260] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:27.322149 7634 ts_tablet_manager.cc:1428] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:54:27.322870 7343 tablet_copy_service.cc:342] P b3b91d8793f540749195348e46332124: Request end of tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-690913b2fc834a7bafecefcb9b823cab received from {username='slave'} at 127.0.0.1:57854
I20250626 01:54:27.322989 7343 tablet_copy_service.cc:434] P b3b91d8793f540749195348e46332124: ending tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-690913b2fc834a7bafecefcb9b823cab on tablet 690913b2fc834a7bafecefcb9b823cab with peer 882d449e16bd4578ab034cc8c7e0e3c4
I20250626 01:54:27.336005 7634 ts_tablet_manager.cc:927] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Initiating tablet copy from peer 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:27.336603 7634 tablet_copy_client.cc:323] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:37253
I20250626 01:54:27.337316 7269 tablet_copy_service.cc:140] P 43874288b35e4f85adc362683201eb19: Received BeginTabletCopySession request for tablet 255285c177a147eabf811f0d749903dc from peer 882d449e16bd4578ab034cc8c7e0e3c4 ({username='slave'} at 127.0.0.1:33708)
I20250626 01:54:27.337474 7269 tablet_copy_service.cc:161] P 43874288b35e4f85adc362683201eb19: Beginning new tablet copy session on tablet 255285c177a147eabf811f0d749903dc from peer 882d449e16bd4578ab034cc8c7e0e3c4 at {username='slave'} at 127.0.0.1:33708: session id = 882d449e16bd4578ab034cc8c7e0e3c4-255285c177a147eabf811f0d749903dc
I20250626 01:54:27.339715 7269 tablet_copy_source_session.cc:215] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:27.341091 7634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255285c177a147eabf811f0d749903dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:27.345609 7634 tablet_copy_client.cc:806] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:27.345870 7634 tablet_copy_client.cc:670] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:27.347409 7634 tablet_copy_client.cc:538] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:27.350171 7634 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap starting.
I20250626 01:54:27.355172 7634 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:27.355484 7634 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap complete.
I20250626 01:54:27.355728 7634 ts_tablet_manager.cc:1397] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250626 01:54:27.356380 7634 raft_consensus.cc:357] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:27.356539 7634 raft_consensus.cc:738] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 882d449e16bd4578ab034cc8c7e0e3c4, State: Initialized, Role: LEARNER
I20250626 01:54:27.356720 7634 consensus_queue.cc:260] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:27.357170 7634 ts_tablet_manager.cc:1428] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250626 01:54:27.357980 7269 tablet_copy_service.cc:342] P 43874288b35e4f85adc362683201eb19: Request end of tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-255285c177a147eabf811f0d749903dc received from {username='slave'} at 127.0.0.1:33708
I20250626 01:54:27.358072 7269 tablet_copy_service.cc:434] P 43874288b35e4f85adc362683201eb19: ending tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-255285c177a147eabf811f0d749903dc on tablet 255285c177a147eabf811f0d749903dc with peer 882d449e16bd4578ab034cc8c7e0e3c4
I20250626 01:54:27.460837 7258 consensus_queue.cc:237] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:27.462405 7594 raft_consensus.cc:1273] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:27.462405 7333 raft_consensus.cc:1273] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:27.462961 7647 consensus_queue.cc:1035] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:27.463193 7646 consensus_queue.cc:1035] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:27.465520 7384 raft_consensus.cc:2953] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.466113 7333 raft_consensus.cc:2953] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.466123 7594 raft_consensus.cc:2953] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.470310 7066 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 26292b94e4ad4a6db496def22e462011 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:27.470906 7079 catalog_manager.cc:5582] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 reported cstate change: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.474315 7495 tablet_service.cc:1515] Processing DeleteTablet for tablet 26292b94e4ad4a6db496def22e462011 with delete_type TABLET_DATA_TOMBSTONED (TS 882d449e16bd4578ab034cc8c7e0e3c4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43632
I20250626 01:54:27.474668 7632 tablet_replica.cc:331] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: stopping tablet replica
I20250626 01:54:27.474876 7632 raft_consensus.cc:2241] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Raft consensus shutting down.
I20250626 01:54:27.475050 7632 raft_consensus.cc:2270] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Raft consensus is shut down!
I20250626 01:54:27.475745 7632 ts_tablet_manager.cc:1905] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:27.481786 7632 ts_tablet_manager.cc:1918] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250626 01:54:27.481920 7632 log.cc:1199] T 26292b94e4ad4a6db496def22e462011 P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root/wals/26292b94e4ad4a6db496def22e462011
I20250626 01:54:27.482648 7066 catalog_manager.cc:4928] TS 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): tablet 26292b94e4ad4a6db496def22e462011 (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]) successfully deleted
W20250626 01:54:27.483525 7130 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID 882d449e16bd4578ab034cc8c7e0e3c4 is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
W20250626 01:54:27.484941 7130 auto_rebalancer.cc:663] Could not move replica: Illegal state: 255285c177a147eabf811f0d749903dc: source replica e5c18a4bcc584d15934a3c8ab2593fe9 does not have REPLACE attribute set
W20250626 01:54:27.485014 7130 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 255285c177a147eabf811f0d749903dc: source replica e5c18a4bcc584d15934a3c8ab2593fe9 does not have REPLACE attribute set
W20250626 01:54:27.486253 7130 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 26292b94e4ad4a6db496def22e462011, TS b3b91d8793f540749195348e46332124 -> TS 882d449e16bd4578ab034cc8c7e0e3c4 move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
W20250626 01:54:27.486330 7130 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 26292b94e4ad4a6db496def22e462011, TS b3b91d8793f540749195348e46332124 -> TS 882d449e16bd4578ab034cc8c7e0e3c4 move failed, destination replica disappeared from tablet's Raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
W20250626 01:54:27.487668 7130 auto_rebalancer.cc:663] Could not move replica: Illegal state: 690913b2fc834a7bafecefcb9b823cab: source replica 43874288b35e4f85adc362683201eb19 does not have REPLACE attribute set
W20250626 01:54:27.487736 7130 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 690913b2fc834a7bafecefcb9b823cab: source replica 43874288b35e4f85adc362683201eb19 does not have REPLACE attribute set
I20250626 01:54:27.646255 7333 consensus_queue.cc:237] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:27.648216 7258 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:27.648216 7594 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:27.648860 7649 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:27.649267 7650 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:27.651437 7650 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.652354 7258 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.652408 7594 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.655470 7069 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 690913b2fc834a7bafecefcb9b823cab with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:27.657042 7081 catalog_manager.cc:5582] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.660670 7495 tablet_service.cc:1515] Processing DeleteTablet for tablet 690913b2fc834a7bafecefcb9b823cab with delete_type TABLET_DATA_TOMBSTONED (TS 882d449e16bd4578ab034cc8c7e0e3c4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43632
I20250626 01:54:27.661046 7632 tablet_replica.cc:331] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: stopping tablet replica
I20250626 01:54:27.661248 7632 raft_consensus.cc:2241] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Raft consensus shutting down.
I20250626 01:54:27.661475 7632 raft_consensus.cc:2270] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Raft consensus is shut down!
I20250626 01:54:27.662304 7632 ts_tablet_manager.cc:1905] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:27.670135 7632 ts_tablet_manager.cc:1918] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250626 01:54:27.670295 7632 log.cc:1199] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root/wals/690913b2fc834a7bafecefcb9b823cab
I20250626 01:54:27.671054 7066 catalog_manager.cc:4928] TS 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): tablet 690913b2fc834a7bafecefcb9b823cab (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]) successfully deleted
I20250626 01:54:27.712842 7258 consensus_queue.cc:237] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } }
I20250626 01:54:27.714622 7594 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:27.714627 7333 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250626 01:54:27.715224 7646 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:27.715459 7384 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250626 01:54:27.717734 7647 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.718482 7333 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.720296 7594 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.720880 7066 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 255285c177a147eabf811f0d749903dc with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:27.722894 7079 catalog_manager.cc:5582] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 reported cstate change: config changed from index 2 to 3, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } }
I20250626 01:54:27.726306 7495 tablet_service.cc:1515] Processing DeleteTablet for tablet 255285c177a147eabf811f0d749903dc with delete_type TABLET_DATA_TOMBSTONED (TS 882d449e16bd4578ab034cc8c7e0e3c4 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:43632
I20250626 01:54:27.726768 7632 tablet_replica.cc:331] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: stopping tablet replica
I20250626 01:54:27.726949 7632 raft_consensus.cc:2241] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Raft consensus shutting down.
I20250626 01:54:27.727137 7632 raft_consensus.cc:2270] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Raft consensus is shut down!
I20250626 01:54:27.727790 7632 ts_tablet_manager.cc:1905] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:27.732224 7632 ts_tablet_manager.cc:1918] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250626 01:54:27.732380 7632 log.cc:1199] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-3-root/wals/255285c177a147eabf811f0d749903dc
I20250626 01:54:27.733189 7066 catalog_manager.cc:4928] TS 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): tablet 255285c177a147eabf811f0d749903dc (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]) successfully deleted
I20250626 01:54:28.491850 7258 consensus_queue.cc:237] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:28.494024 7333 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250626 01:54:28.494205 7222 consensus_peers.cc:487] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:28.494267 7594 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:28.494617 7646 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:28.494835 7652 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:28.496913 7651 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.497452 7594 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.500579 7333 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.501451 7081 catalog_manager.cc:5582] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:28.526842 7333 consensus_queue.cc:237] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:28.529000 7258 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:28.529021 7594 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250626 01:54:28.529186 7296 consensus_peers.cc:487] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:28.529531 7660 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:28.529836 7650 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:28.531922 7660 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.533011 7258 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.533169 7594 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.537792 7079 catalog_manager.cc:5582] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 reported cstate change: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.539163 7333 consensus_queue.cc:237] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:28.547446 7595 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250626 01:54:28.548286 7258 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20250626 01:54:28.548345 7296 consensus_peers.cc:487] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 -> Peer 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68:36707): Couldn't send request to peer 882d449e16bd4578ab034cc8c7e0e3c4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fce1c61b02da4ff390f35254b24356c7. This is attempt 1: this message will repeat every 5th retry.
I20250626 01:54:28.548796 7650 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:28.549074 7661 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250626 01:54:28.551801 7663 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.555248 7595 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.555248 7258 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } } }
I20250626 01:54:28.556444 7080 catalog_manager.cc:5582] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 reported cstate change: config changed from index 3 to 4, NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:28.875993 7667 ts_tablet_manager.cc:927] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Initiating tablet copy from peer 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253)
I20250626 01:54:28.876722 7667 tablet_copy_client.cc:323] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:37253
I20250626 01:54:28.877480 7269 tablet_copy_service.cc:140] P 43874288b35e4f85adc362683201eb19: Received BeginTabletCopySession request for tablet 255285c177a147eabf811f0d749903dc from peer 882d449e16bd4578ab034cc8c7e0e3c4 ({username='slave'} at 127.0.0.1:33708)
I20250626 01:54:28.877732 7269 tablet_copy_service.cc:161] P 43874288b35e4f85adc362683201eb19: Beginning new tablet copy session on tablet 255285c177a147eabf811f0d749903dc from peer 882d449e16bd4578ab034cc8c7e0e3c4 at {username='slave'} at 127.0.0.1:33708: session id = 882d449e16bd4578ab034cc8c7e0e3c4-255285c177a147eabf811f0d749903dc
I20250626 01:54:28.880101 7269 tablet_copy_source_session.cc:215] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:28.881050 7667 ts_tablet_manager.cc:1905] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250626 01:54:28.886907 7667 ts_tablet_manager.cc:1918] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20250626 01:54:28.887243 7667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 255285c177a147eabf811f0d749903dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:28.890236 7667 tablet_copy_client.cc:806] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:28.890547 7667 tablet_copy_client.cc:670] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:28.892144 7667 tablet_copy_client.cc:538] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:28.895300 7667 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap starting.
I20250626 01:54:28.902439 7667 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:28.902832 7667 tablet_bootstrap.cc:492] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap complete.
I20250626 01:54:28.903100 7667 ts_tablet_manager.cc:1397] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250626 01:54:28.903861 7667 raft_consensus.cc:357] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:28.904103 7667 raft_consensus.cc:738] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 882d449e16bd4578ab034cc8c7e0e3c4, State: Initialized, Role: LEARNER
I20250626 01:54:28.904287 7667 consensus_queue.cc:260] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:28.905544 7667 ts_tablet_manager.cc:1428] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20250626 01:54:28.906155 7269 tablet_copy_service.cc:342] P 43874288b35e4f85adc362683201eb19: Request end of tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-255285c177a147eabf811f0d749903dc received from {username='slave'} at 127.0.0.1:33708
I20250626 01:54:28.906281 7269 tablet_copy_service.cc:434] P 43874288b35e4f85adc362683201eb19: ending tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-255285c177a147eabf811f0d749903dc on tablet 255285c177a147eabf811f0d749903dc with peer 882d449e16bd4578ab034cc8c7e0e3c4
I20250626 01:54:28.995299 7667 ts_tablet_manager.cc:927] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Initiating tablet copy from peer b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:28.995924 7667 tablet_copy_client.cc:323] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:36473
I20250626 01:54:28.996479 7343 tablet_copy_service.cc:140] P b3b91d8793f540749195348e46332124: Received BeginTabletCopySession request for tablet 690913b2fc834a7bafecefcb9b823cab from peer 882d449e16bd4578ab034cc8c7e0e3c4 ({username='slave'} at 127.0.0.1:57854)
I20250626 01:54:28.996608 7343 tablet_copy_service.cc:161] P b3b91d8793f540749195348e46332124: Beginning new tablet copy session on tablet 690913b2fc834a7bafecefcb9b823cab from peer 882d449e16bd4578ab034cc8c7e0e3c4 at {username='slave'} at 127.0.0.1:57854: session id = 882d449e16bd4578ab034cc8c7e0e3c4-690913b2fc834a7bafecefcb9b823cab
I20250626 01:54:28.998867 7343 tablet_copy_source_session.cc:215] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:28.999764 7667 ts_tablet_manager.cc:1905] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250626 01:54:29.005249 7667 ts_tablet_manager.cc:1918] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20250626 01:54:29.005550 7667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 690913b2fc834a7bafecefcb9b823cab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:29.008427 7667 tablet_copy_client.cc:806] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:29.008802 7667 tablet_copy_client.cc:670] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:29.010461 7667 tablet_copy_client.cc:538] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:29.014796 7667 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap starting.
I20250626 01:54:29.021399 7667 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:29.021790 7667 tablet_bootstrap.cc:492] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap complete.
I20250626 01:54:29.022054 7667 ts_tablet_manager.cc:1397] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20250626 01:54:29.022857 7667 raft_consensus.cc:357] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:29.023032 7667 raft_consensus.cc:738] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 882d449e16bd4578ab034cc8c7e0e3c4, State: Initialized, Role: LEARNER
I20250626 01:54:29.023224 7667 consensus_queue.cc:260] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:29.023600 7667 ts_tablet_manager.cc:1428] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250626 01:54:29.024245 7343 tablet_copy_service.cc:342] P b3b91d8793f540749195348e46332124: Request end of tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-690913b2fc834a7bafecefcb9b823cab received from {username='slave'} at 127.0.0.1:57854
I20250626 01:54:29.024353 7343 tablet_copy_service.cc:434] P b3b91d8793f540749195348e46332124: ending tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-690913b2fc834a7bafecefcb9b823cab on tablet 690913b2fc834a7bafecefcb9b823cab with peer 882d449e16bd4578ab034cc8c7e0e3c4
I20250626 01:54:29.162388 7667 ts_tablet_manager.cc:927] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: Initiating tablet copy from peer b3b91d8793f540749195348e46332124 (127.0.103.67:36473)
I20250626 01:54:29.163059 7667 tablet_copy_client.cc:323] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:36473
I20250626 01:54:29.163625 7343 tablet_copy_service.cc:140] P b3b91d8793f540749195348e46332124: Received BeginTabletCopySession request for tablet fce1c61b02da4ff390f35254b24356c7 from peer 882d449e16bd4578ab034cc8c7e0e3c4 ({username='slave'} at 127.0.0.1:57854)
I20250626 01:54:29.163734 7343 tablet_copy_service.cc:161] P b3b91d8793f540749195348e46332124: Beginning new tablet copy session on tablet fce1c61b02da4ff390f35254b24356c7 from peer 882d449e16bd4578ab034cc8c7e0e3c4 at {username='slave'} at 127.0.0.1:57854: session id = 882d449e16bd4578ab034cc8c7e0e3c4-fce1c61b02da4ff390f35254b24356c7
I20250626 01:54:29.166142 7343 tablet_copy_source_session.cc:215] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: Tablet Copy: opened 0 blocks and 1 log segments
I20250626 01:54:29.167181 7667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fce1c61b02da4ff390f35254b24356c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250626 01:54:29.172281 7667 tablet_copy_client.cc:806] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 0 data blocks...
I20250626 01:54:29.172573 7667 tablet_copy_client.cc:670] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Starting download of 1 WAL segments...
I20250626 01:54:29.173970 7667 tablet_copy_client.cc:538] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250626 01:54:29.176649 7667 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap starting.
I20250626 01:54:29.182395 7667 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250626 01:54:29.182754 7667 tablet_bootstrap.cc:492] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: Bootstrap complete.
I20250626 01:54:29.183020 7667 ts_tablet_manager.cc:1397] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250626 01:54:29.183738 7667 raft_consensus.cc:357] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:29.183975 7667 raft_consensus.cc:738] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 882d449e16bd4578ab034cc8c7e0e3c4, State: Initialized, Role: LEARNER
I20250626 01:54:29.184183 7667 consensus_queue.cc:260] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: true } }
I20250626 01:54:29.184669 7667 ts_tablet_manager.cc:1428] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20250626 01:54:29.185324 7343 tablet_copy_service.cc:342] P b3b91d8793f540749195348e46332124: Request end of tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-fce1c61b02da4ff390f35254b24356c7 received from {username='slave'} at 127.0.0.1:57854
I20250626 01:54:29.185448 7343 tablet_copy_service.cc:434] P b3b91d8793f540749195348e46332124: ending tablet copy session 882d449e16bd4578ab034cc8c7e0e3c4-fce1c61b02da4ff390f35254b24356c7 on tablet fce1c61b02da4ff390f35254b24356c7 with peer 882d449e16bd4578ab034cc8c7e0e3c4
I20250626 01:54:29.424714 7505 raft_consensus.cc:1215] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250626 01:54:29.545979 7505 raft_consensus.cc:1215] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250626 01:54:29.617825 7505 raft_consensus.cc:1215] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20250626 01:54:29.861546 7675 raft_consensus.cc:1062] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: attempting to promote NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 to VOTER
I20250626 01:54:29.862286 7675 consensus_queue.cc:237] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } }
I20250626 01:54:29.864544 7505 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:29.864936 7595 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:29.865276 7332 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:29.865653 7675 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:29.865928 7680 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:29.866168 7676 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:29.869499 7675 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.870333 7333 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.873147 7595 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.873369 7505 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.876852 7080 catalog_manager.cc:5582] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:54:29.890820 7259 consensus_queue.cc:237] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } }
I20250626 01:54:29.892572 7505 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 01:54:29.893595 7595 raft_consensus.cc:1273] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer 43874288b35e4f85adc362683201eb19: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 01:54:29.893743 7676 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:54:29.894157 7653 consensus_queue.cc:1035] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:54:29.896163 7676 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3b91d8793f540749195348e46332124 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.896853 7595 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3b91d8793f540749195348e46332124 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.899128 7066 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 255285c177a147eabf811f0d749903dc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:29.901160 7082 catalog_manager.cc:5582] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 reported cstate change: config changed from index 5 to 6, VOTER b3b91d8793f540749195348e46332124 (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "43874288b35e4f85adc362683201eb19" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250626 01:54:29.902827 7505 raft_consensus.cc:2953] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER b3b91d8793f540749195348e46332124 (127.0.103.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:29.904954 7321 tablet_service.cc:1515] Processing DeleteTablet for tablet 255285c177a147eabf811f0d749903dc with delete_type TABLET_DATA_TOMBSTONED (TS b3b91d8793f540749195348e46332124 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51580
I20250626 01:54:29.905680 7689 tablet_replica.cc:331] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:29.905979 7689 raft_consensus.cc:2241] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:29.906414 7689 raft_consensus.cc:2270] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:29.908895 7689 ts_tablet_manager.cc:1905] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:29.914307 7689 ts_tablet_manager.cc:1918] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250626 01:54:29.914454 7689 log.cc:1199] T 255285c177a147eabf811f0d749903dc P b3b91d8793f540749195348e46332124: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-2-root/wals/255285c177a147eabf811f0d749903dc
I20250626 01:54:29.915118 7069 catalog_manager.cc:4928] TS b3b91d8793f540749195348e46332124 (127.0.103.67:36473): tablet 255285c177a147eabf811f0d749903dc (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]) successfully deleted
I20250626 01:54:30.024087 7650 raft_consensus.cc:1062] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: attempting to promote NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 to VOTER
I20250626 01:54:30.024909 7650 consensus_queue.cc:237] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } }
I20250626 01:54:30.027293 7258 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:30.027319 7595 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:30.027324 7505 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:30.028002 7650 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:30.028371 7678 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:30.028627 7677 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:30.045709 7678 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.046458 7505 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.046437 7258 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.051224 7082 catalog_manager.cc:5582] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 reported cstate change: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.053853 7595 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } attrs { replace: true } } peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.056180 7333 consensus_queue.cc:237] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } }
I20250626 01:54:30.058602 7505 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 01:54:30.059451 7258 raft_consensus.cc:1273] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 01:54:30.059988 7677 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:54:30.060233 7678 consensus_queue.cc:1035] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:54:30.062592 7677 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.063478 7504 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.063868 7258 raft_consensus.cc:2953] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.066335 7069 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 690913b2fc834a7bafecefcb9b823cab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:30.067960 7081 catalog_manager.cc:5582] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 reported cstate change: config changed from index 5 to 6, VOTER e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:54:30.071566 7581 tablet_service.cc:1515] Processing DeleteTablet for tablet 690913b2fc834a7bafecefcb9b823cab with delete_type TABLET_DATA_TOMBSTONED (TS e5c18a4bcc584d15934a3c8ab2593fe9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46694
I20250626 01:54:30.072574 7695 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:30.072882 7695 raft_consensus.cc:2241] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.073174 7695 raft_consensus.cc:2270] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.073956 7695 ts_tablet_manager.cc:1905] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:30.079320 7695 ts_tablet_manager.cc:1918] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250626 01:54:30.079475 7695 log.cc:1199] T 690913b2fc834a7bafecefcb9b823cab P e5c18a4bcc584d15934a3c8ab2593fe9: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-0-root/wals/690913b2fc834a7bafecefcb9b823cab
I20250626 01:54:30.080945 7066 catalog_manager.cc:4928] TS e5c18a4bcc584d15934a3c8ab2593fe9 (127.0.103.65:32973): tablet 690913b2fc834a7bafecefcb9b823cab (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]) successfully deleted
I20250626 01:54:30.191689 7678 raft_consensus.cc:1062] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: attempting to promote NON_VOTER 882d449e16bd4578ab034cc8c7e0e3c4 to VOTER
I20250626 01:54:30.192430 7678 consensus_queue.cc:237] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } }
I20250626 01:54:30.194732 7595 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:30.194797 7504 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 LEARNER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:30.194870 7258 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20250626 01:54:30.195274 7694 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:30.195657 7678 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:30.195901 7677 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250626 01:54:30.198982 7678 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.199803 7258 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.200814 7595 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.203136 7504 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.204972 7079 catalog_manager.cc:5582] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 reported cstate change: config changed from index 4 to 5, 882d449e16bd4578ab034cc8c7e0e3c4 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "43874288b35e4f85adc362683201eb19" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250626 01:54:30.211176 7333 consensus_queue.cc:237] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } }
I20250626 01:54:30.213122 7504 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 01:54:30.213469 7595 raft_consensus.cc:1273] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Refusing update from remote peer b3b91d8793f540749195348e46332124: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20250626 01:54:30.213826 7694 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:54:30.215031 7694 consensus_queue.cc:1035] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250626 01:54:30.216542 7700 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43874288b35e4f85adc362683201eb19 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.217265 7504 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43874288b35e4f85adc362683201eb19 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.218166 7595 raft_consensus.cc:2953] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 43874288b35e4f85adc362683201eb19 (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.221316 7069 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet fce1c61b02da4ff390f35254b24356c7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250626 01:54:30.221936 7080 catalog_manager.cc:5582] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 reported cstate change: config changed from index 5 to 6, VOTER 43874288b35e4f85adc362683201eb19 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "b3b91d8793f540749195348e46332124" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b3b91d8793f540749195348e46332124" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36473 } } peers { permanent_uuid: "e5c18a4bcc584d15934a3c8ab2593fe9" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 32973 } } peers { permanent_uuid: "882d449e16bd4578ab034cc8c7e0e3c4" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36707 } attrs { promote: false } } }
I20250626 01:54:30.225450 7249 tablet_service.cc:1515] Processing DeleteTablet for tablet fce1c61b02da4ff390f35254b24356c7 with delete_type TABLET_DATA_TOMBSTONED (TS 43874288b35e4f85adc362683201eb19 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:46408
I20250626 01:54:30.226125 7701 tablet_replica.cc:331] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:30.226317 7701 raft_consensus.cc:2241] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.226635 7701 raft_consensus.cc:2270] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.227433 7701 ts_tablet_manager.cc:1905] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250626 01:54:30.231992 7701 ts_tablet_manager.cc:1918] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20250626 01:54:30.232129 7701 log.cc:1199] T fce1c61b02da4ff390f35254b24356c7 P 43874288b35e4f85adc362683201eb19: Deleting WAL directory at /tmp/dist-test-taskGCfI_O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1750902684575999-413-0/minicluster-data/ts-1-root/wals/fce1c61b02da4ff390f35254b24356c7
I20250626 01:54:30.232879 7066 catalog_manager.cc:4928] TS 43874288b35e4f85adc362683201eb19 (127.0.103.66:37253): tablet fce1c61b02da4ff390f35254b24356c7 (table test-workload [id=53b9c883f3c5440a8bcbbb79215061ae]) successfully deleted
I20250626 01:54:30.786427 413 tablet_server.cc:178] TabletServer@127.0.103.65:32973 shutting down...
I20250626 01:54:30.794824 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:30.795080 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:30.795259 413 raft_consensus.cc:2241] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.795581 413 raft_consensus.cc:2270] T 255285c177a147eabf811f0d749903dc P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.796422 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:30.796581 413 raft_consensus.cc:2241] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.796782 413 raft_consensus.cc:2270] T 26292b94e4ad4a6db496def22e462011 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.797287 413 tablet_replica.cc:331] stopping tablet replica
I20250626 01:54:30.797433 413 raft_consensus.cc:2241] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.797710 413 raft_consensus.cc:2270] T fce1c61b02da4ff390f35254b24356c7 P e5c18a4bcc584d15934a3c8ab2593fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.812825 413 tablet_server.cc:195] TabletServer@127.0.103.65:32973 shutdown complete.
I20250626 01:54:30.817474 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20250626 01:54:30.837366 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:30.837710 413 tablet_replica.cc:331] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:30.837891 413 raft_consensus.cc:2241] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.838178 413 raft_consensus.cc:2270] T 690913b2fc834a7bafecefcb9b823cab P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.838940 413 tablet_replica.cc:331] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:30.839051 413 raft_consensus.cc:2241] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:30.839314 413 raft_consensus.cc:2270] T 26292b94e4ad4a6db496def22e462011 P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.839741 413 tablet_replica.cc:331] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19: stopping tablet replica
I20250626 01:54:30.839848 413 raft_consensus.cc:2241] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:30.840107 413 raft_consensus.cc:2270] T 255285c177a147eabf811f0d749903dc P 43874288b35e4f85adc362683201eb19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.855057 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20250626 01:54:30.860013 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20250626 01:54:30.869395 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:30.869709 413 tablet_replica.cc:331] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:30.869894 413 raft_consensus.cc:2241] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:30.870232 413 raft_consensus.cc:2270] T fce1c61b02da4ff390f35254b24356c7 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.870991 413 tablet_replica.cc:331] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:30.871109 413 raft_consensus.cc:2241] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.871311 413 raft_consensus.cc:2270] T 26292b94e4ad4a6db496def22e462011 P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.871758 413 tablet_replica.cc:331] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124: stopping tablet replica
I20250626 01:54:30.871948 413 raft_consensus.cc:2241] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:30.872205 413 raft_consensus.cc:2270] T 690913b2fc834a7bafecefcb9b823cab P b3b91d8793f540749195348e46332124 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.886857 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20250626 01:54:30.891350 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20250626 01:54:30.899966 413 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250626 01:54:30.900243 413 tablet_replica.cc:331] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4: stopping tablet replica
I20250626 01:54:30.900424 413 raft_consensus.cc:2241] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.900694 413 raft_consensus.cc:2270] T fce1c61b02da4ff390f35254b24356c7 P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.901484 413 tablet_replica.cc:331] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4: stopping tablet replica
I20250626 01:54:30.901639 413 raft_consensus.cc:2241] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.901835 413 raft_consensus.cc:2270] T 690913b2fc834a7bafecefcb9b823cab P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.902630 413 tablet_replica.cc:331] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4: stopping tablet replica
I20250626 01:54:30.902777 413 raft_consensus.cc:2241] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250626 01:54:30.902971 413 raft_consensus.cc:2270] T 255285c177a147eabf811f0d749903dc P 882d449e16bd4578ab034cc8c7e0e3c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.925846 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20250626 01:54:30.929517 413 master.cc:561] Master@127.0.103.126:41555 shutting down...
I20250626 01:54:30.936674 413 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 1 LEADER]: Raft consensus shutting down.
I20250626 01:54:30.936937 413 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250626 01:54:30.937036 413 tablet_replica.cc:331] T 00000000000000000000000000000000 P 08194e68dffb4b5299ee517ac4649748: stopping tablet replica
I20250626 01:54:30.951052 413 master.cc:583] Master@127.0.103.126:41555 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (7559 ms)
[----------] 14 tests from AutoRebalancerTest (186318 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (186318 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20250626 01:54:30.965942 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5260: suppressed but not reported on 13 messages since previous log ~4 seconds ago
I20250626 01:54:30.966040 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250626 01:54:30.966084 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250626 01:54:30.966128 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5112: suppressed but not reported on 1 messages since previous log ~72 seconds ago
I20250626 01:54:30.966168 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4671: suppressed but not reported on 12 messages since previous log ~4 seconds ago
I20250626 01:54:30.966229 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~172 seconds ago
I20250626 01:54:30.966289 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 51 messages since previous log ~17 seconds ago
I20250626 01:54:30.966325 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 32 messages since previous log ~5 seconds ago