[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251026 08:51:09.233034 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:33569,127.0.103.125:41243,127.0.103.124:41525
I20251026 08:51:09.234969 413 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251026 08:51:09.236222 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:09.247056 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.247260 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.247622 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.248596 413 server_base.cc:1047] running on GCE node
I20251026 08:51:09.249341 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:09.249495 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:09.249567 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468669249550 us; error 0 us; skew 500 ppm
I20251026 08:51:09.249822 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:09.251972 413 webserver.cc:492] Webserver started at http://127.0.103.126:37183/ using document root <none> and password file <none>
I20251026 08:51:09.252604 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:09.252696 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:09.253011 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:09.255944 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "81d02c4294534063aef498aa325649a6"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.261189 413 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251026 08:51:09.264145 426 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.265403 413 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251026 08:51:09.265553 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "81d02c4294534063aef498aa325649a6"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.265709 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:09.286753 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:09.287416 413 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251026 08:51:09.287674 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:09.307904 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:33569
I20251026 08:51:09.307935 477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:33569 every 8 connection(s)
I20251026 08:51:09.309908 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:09.312482 480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.312567 481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.313836 478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 08:51:09.314147 483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.319070 413 server_base.cc:1047] running on GCE node
I20251026 08:51:09.319317 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:09.319387 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:09.319464 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468669319427 us; error 0 us; skew 500 ppm
I20251026 08:51:09.319614 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:09.320876 413 webserver.cc:492] Webserver started at http://127.0.103.125:43235/ using document root <none> and password file <none>
I20251026 08:51:09.321107 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:09.321205 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:09.321337 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:09.321906 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root/instance:
uuid: "db2b1a7b08c249a89892efe6e84e7816"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.324368 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251026 08:51:09.324513 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:09.325845 488 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.326231 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:09.326357 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root
uuid: "db2b1a7b08c249a89892efe6e84e7816"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.326493 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:09.341825 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
W20251026 08:51:09.342954 429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.125:41243: connect: Connection refused (error 111)
W20251026 08:51:09.344453 478 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.103.125:41243: Network error: Client connection negotiation failed: client connection to 127.0.103.125:41243: connect: Connection refused (error 111)
I20251026 08:51:09.346920 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:09.347427 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:09.363413 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:41243
I20251026 08:51:09.363924 542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.125:41243 every 8 connection(s)
I20251026 08:51:09.365115 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 08:51:09.365149 543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20251026 08:51:09.367841 545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.367976 546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.368947 548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.369319 413 server_base.cc:1047] running on GCE node
I20251026 08:51:09.369431 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:09.369619 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:09.369700 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:09.369774 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468669369737 us; error 0 us; skew 500 ppm
I20251026 08:51:09.369951 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:09.372534 413 webserver.cc:492] Webserver started at http://127.0.103.124:34851/ using document root <none> and password file <none>
I20251026 08:51:09.372799 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:09.372896 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:09.373050 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:09.373764 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root/instance:
uuid: "8278e8fef9c6406b84c17855fa85646d"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.376492 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:09.376637 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 08:51:09.378693 555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.379078 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:09.379180 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root
uuid: "8278e8fef9c6406b84c17855fa85646d"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.379307 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:09.383358 543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
W20251026 08:51:09.385175 543 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.103.124:41525: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41525: connect: Connection refused (error 111)
I20251026 08:51:09.388126 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:09.388562 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:09.393745 478 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } attempt: 1
I20251026 08:51:09.401782 478 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
W20251026 08:51:09.403373 478 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.103.124:41525: Network error: Client connection negotiation failed: client connection to 127.0.103.124:41525: connect: Connection refused (error 111)
I20251026 08:51:09.404776 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:41525
I20251026 08:51:09.404839 607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.124:41525 every 8 connection(s)
I20251026 08:51:09.406237 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:51:09.406550 608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:09.408648 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:09.413722 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:09.418344 608 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:09.427524 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: Bootstrap starting.
I20251026 08:51:09.430653 608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:09.431685 608 log.cc:826] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: Log is configured to *not* fsync() on all Append() calls
I20251026 08:51:09.433753 608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: No bootstrap required, opened a new log
I20251026 08:51:09.437371 543 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } attempt: 1
I20251026 08:51:09.439124 608 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.439388 608 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:09.439495 608 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8278e8fef9c6406b84c17855fa85646d, State: Initialized, Role: FOLLOWER
I20251026 08:51:09.440121 608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.442440 608 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:09.441403 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.442571 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.445528 543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: Bootstrap starting.
I20251026 08:51:09.446578 478 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } attempt: 1
I20251026 08:51:09.447403 543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:09.450507 543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: No bootstrap required, opened a new log
I20251026 08:51:09.451331 543 raft_consensus.cc:359] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.451537 543 raft_consensus.cc:385] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:09.451627 543 raft_consensus.cc:740] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db2b1a7b08c249a89892efe6e84e7816, State: Initialized, Role: FOLLOWER
I20251026 08:51:09.452028 543 consensus_queue.cc:260] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
W20251026 08:51:09.456218 624 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 08:51:09.456344 624 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 08:51:09.456641 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: Bootstrap starting.
I20251026 08:51:09.458909 478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:09.458925 627 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.459283 627 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.459471 543 sys_catalog.cc:565] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:09.461006 478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: No bootstrap required, opened a new log
I20251026 08:51:09.461978 478 raft_consensus.cc:359] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.462605 478 raft_consensus.cc:385] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:09.462713 478 raft_consensus.cc:740] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d02c4294534063aef498aa325649a6, State: Initialized, Role: FOLLOWER
I20251026 08:51:09.462962 478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.466861 640 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251026 08:51:09.466830 635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.467111 478 sys_catalog.cc:565] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:09.467150 635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.471657 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20251026 08:51:09.471776 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20251026 08:51:09.472671 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:09.472983 651 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 08:51:09.473073 651 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251026 08:51:09.476336 652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.476816 653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.478200 655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.478579 413 server_base.cc:1047] running on GCE node
I20251026 08:51:09.478791 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:09.478864 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:09.478919 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468669478901 us; error 0 us; skew 500 ppm
I20251026 08:51:09.479079 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:09.480250 413 webserver.cc:492] Webserver started at http://127.0.103.65:43969/ using document root <none> and password file <none>
I20251026 08:51:09.480459 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:09.480535 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:09.480691 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:09.481235 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "725a688880b7412e87454cf66de08999"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.483543 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 08:51:09.485538 660 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.485921 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:09.486030 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "725a688880b7412e87454cf66de08999"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.486163 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:09.509488 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:09.510179 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:09.511087 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:09.512663 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:09.513243 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.513368 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:09.513443 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.533079 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:36409
I20251026 08:51:09.533098 722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:36409 every 8 connection(s)
W20251026 08:51:09.539422 432 tablet.cc:2378] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 08:51:09.564795 724 heartbeater.cc:344] Connected to a master server at 127.0.103.126:33569
I20251026 08:51:09.564916 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 08:51:09.565090 724 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.566210 724 heartbeater.cc:507] Master 127.0.103.126:33569 requested a full tablet report, sending...
I20251026 08:51:09.571684 725 heartbeater.cc:344] Connected to a master server at 127.0.103.125:41243
I20251026 08:51:09.571830 725 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.572422 725 heartbeater.cc:507] Master 127.0.103.125:41243 requested a full tablet report, sending...
W20251026 08:51:09.572985 730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.573910 443 ts_manager.cc:194] Registered new tserver with Master: 725a688880b7412e87454cf66de08999 (127.0.103.65:36409)
I20251026 08:51:09.574612 508 ts_manager.cc:194] Registered new tserver with Master: 725a688880b7412e87454cf66de08999 (127.0.103.65:36409)
I20251026 08:51:09.576121 723 heartbeater.cc:344] Connected to a master server at 127.0.103.124:41525
I20251026 08:51:09.576248 723 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.577096 723 heartbeater.cc:507] Master 127.0.103.124:41525 requested a full tablet report, sending...
W20251026 08:51:09.577178 731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.578440 733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.578478 573 ts_manager.cc:194] Registered new tserver with Master: 725a688880b7412e87454cf66de08999 (127.0.103.65:36409)
I20251026 08:51:09.583112 413 server_base.cc:1047] running on GCE node
I20251026 08:51:09.583474 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:09.583540 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:09.583616 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468669583579 us; error 0 us; skew 500 ppm
I20251026 08:51:09.583777 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:09.585312 413 webserver.cc:492] Webserver started at http://127.0.103.66:38379/ using document root <none> and password file <none>
I20251026 08:51:09.585536 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:09.585630 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:09.585755 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:09.586426 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "d49db780211440118377f5cb26102527"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.589342 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20251026 08:51:09.591074 738 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.591437 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:09.591552 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "d49db780211440118377f5cb26102527"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.591675 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:09.619041 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:09.619563 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:09.620167 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:09.621155 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:09.621228 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.621349 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:09.621412 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.642390 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:35895
I20251026 08:51:09.642462 800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:35895 every 8 connection(s)
I20251026 08:51:09.655464 802 heartbeater.cc:344] Connected to a master server at 127.0.103.126:33569
I20251026 08:51:09.655643 802 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.656641 802 heartbeater.cc:507] Master 127.0.103.126:33569 requested a full tablet report, sending...
I20251026 08:51:09.657868 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 08:51:09.658309 443 ts_manager.cc:194] Registered new tserver with Master: d49db780211440118377f5cb26102527 (127.0.103.66:35895)
I20251026 08:51:09.659462 801 heartbeater.cc:344] Connected to a master server at 127.0.103.124:41525
I20251026 08:51:09.659626 801 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.660250 801 heartbeater.cc:507] Master 127.0.103.124:41525 requested a full tablet report, sending...
I20251026 08:51:09.661310 573 ts_manager.cc:194] Registered new tserver with Master: d49db780211440118377f5cb26102527 (127.0.103.66:35895)
W20251026 08:51:09.667510 808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.669062 803 heartbeater.cc:344] Connected to a master server at 127.0.103.125:41243
I20251026 08:51:09.669196 803 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.670238 803 heartbeater.cc:507] Master 127.0.103.125:41243 requested a full tablet report, sending...
I20251026 08:51:09.671224 508 ts_manager.cc:194] Registered new tserver with Master: d49db780211440118377f5cb26102527 (127.0.103.66:35895)
W20251026 08:51:09.672667 809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:09.675556 811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:09.675707 413 server_base.cc:1047] running on GCE node
I20251026 08:51:09.676192 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:09.676281 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:09.676349 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468669676327 us; error 0 us; skew 500 ppm
I20251026 08:51:09.676523 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:09.677780 413 webserver.cc:492] Webserver started at http://127.0.103.67:35623/ using document root <none> and password file <none>
I20251026 08:51:09.678016 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:09.678133 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:09.678287 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:09.678879 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "6337e065e933404b8bb90bc96193e77f"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.681370 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:51:09.683020 816 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.683385 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:09.683480 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "6337e065e933404b8bb90bc96193e77f"
format_stamp: "Formatted at 2025-10-26 08:51:09 on dist-test-slave-rd83"
I20251026 08:51:09.683583 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:09.697108 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:09.697660 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:09.698282 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:09.699671 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:09.699760 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.699854 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:09.699929 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:09.720644 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35335
I20251026 08:51:09.720681 878 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35335 every 8 connection(s)
I20251026 08:51:09.732937 879 heartbeater.cc:344] Connected to a master server at 127.0.103.124:41525
I20251026 08:51:09.733084 879 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.734133 879 heartbeater.cc:507] Master 127.0.103.124:41525 requested a full tablet report, sending...
I20251026 08:51:09.735170 573 ts_manager.cc:194] Registered new tserver with Master: 6337e065e933404b8bb90bc96193e77f (127.0.103.67:35335)
I20251026 08:51:09.743369 880 heartbeater.cc:344] Connected to a master server at 127.0.103.126:33569
I20251026 08:51:09.743523 880 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.744225 880 heartbeater.cc:507] Master 127.0.103.126:33569 requested a full tablet report, sending...
I20251026 08:51:09.744571 881 heartbeater.cc:344] Connected to a master server at 127.0.103.125:41243
I20251026 08:51:09.745289 881 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:09.745795 443 ts_manager.cc:194] Registered new tserver with Master: 6337e065e933404b8bb90bc96193e77f (127.0.103.67:35335)
I20251026 08:51:09.746146 881 heartbeater.cc:507] Master 127.0.103.125:41243 requested a full tablet report, sending...
I20251026 08:51:09.747032 508 ts_manager.cc:194] Registered new tserver with Master: 6337e065e933404b8bb90bc96193e77f (127.0.103.67:35335)
I20251026 08:51:09.747839 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.002576974s
I20251026 08:51:09.752877 613 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:09.753119 613 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.754720 613 leader_election.cc:290] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81d02c4294534063aef498aa325649a6 (127.0.103.126:33569), db2b1a7b08c249a89892efe6e84e7816 (127.0.103.125:41243)
I20251026 08:51:09.755336 453 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8278e8fef9c6406b84c17855fa85646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d02c4294534063aef498aa325649a6" is_pre_election: true
I20251026 08:51:09.755388 518 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8278e8fef9c6406b84c17855fa85646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2b1a7b08c249a89892efe6e84e7816" is_pre_election: true
I20251026 08:51:09.755698 453 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8278e8fef9c6406b84c17855fa85646d in term 0.
I20251026 08:51:09.755703 518 raft_consensus.cc:2468] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8278e8fef9c6406b84c17855fa85646d in term 0.
I20251026 08:51:09.756225 558 leader_election.cc:304] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 81d02c4294534063aef498aa325649a6, 8278e8fef9c6406b84c17855fa85646d, db2b1a7b08c249a89892efe6e84e7816; no voters:
I20251026 08:51:09.756698 613 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:09.756871 613 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:09.757009 613 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:09.759689 613 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.760310 613 leader_election.cc:290] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [CANDIDATE]: Term 1 election: Requested vote from peers 81d02c4294534063aef498aa325649a6 (127.0.103.126:33569), db2b1a7b08c249a89892efe6e84e7816 (127.0.103.125:41243)
I20251026 08:51:09.760622 453 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8278e8fef9c6406b84c17855fa85646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81d02c4294534063aef498aa325649a6"
I20251026 08:51:09.760747 518 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8278e8fef9c6406b84c17855fa85646d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db2b1a7b08c249a89892efe6e84e7816"
I20251026 08:51:09.760802 453 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:09.760907 518 raft_consensus.cc:3060] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:09.763958 518 raft_consensus.cc:2468] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8278e8fef9c6406b84c17855fa85646d in term 1.
I20251026 08:51:09.763980 453 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8278e8fef9c6406b84c17855fa85646d in term 1.
I20251026 08:51:09.764453 557 leader_election.cc:304] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81d02c4294534063aef498aa325649a6, 8278e8fef9c6406b84c17855fa85646d; no voters:
I20251026 08:51:09.764750 613 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:09.765209 613 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 1 LEADER]: Becoming Leader. State: Replica: 8278e8fef9c6406b84c17855fa85646d, State: Running, Role: LEADER
I20251026 08:51:09.765794 613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } }
I20251026 08:51:09.769479 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8278e8fef9c6406b84c17855fa85646d. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.770416 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:09.771183 889 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:51:09.773458 889 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:51:09.781693 453 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 1 FOLLOWER]: Refusing update from remote peer 8278e8fef9c6406b84c17855fa85646d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:09.781692 518 raft_consensus.cc:1275] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 1 FOLLOWER]: Refusing update from remote peer 8278e8fef9c6406b84c17855fa85646d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:09.782455 613 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [LEADER]: Connected to new peer: Peer: permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:09.782716 887 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [LEADER]: Connected to new peer: Peer: permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:09.786657 627 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8278e8fef9c6406b84c17855fa85646d. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.786911 627 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.789816 891 mvcc.cc:204] Tried to move back new op lower bound from 7214975671415316480 to 7214975671366389760. Current Snapshot: MvccSnapshot[applied={T|T < 7214975671415316480}]
I20251026 08:51:09.790280 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.790516 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:09.791432 889 catalog_manager.cc:1357] Generated new cluster ID: a0874ce9f7f14163992c32ce892c6494
I20251026 08:51:09.791529 889 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:51:09.800207 627 sys_catalog.cc:455] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.800361 627 sys_catalog.cc:458] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.806572 635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8278e8fef9c6406b84c17855fa85646d. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.806777 635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.810598 635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.810839 635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:09.812001 887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8278e8fef9c6406b84c17855fa85646d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81d02c4294534063aef498aa325649a6" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 33569 } } peers { permanent_uuid: "db2b1a7b08c249a89892efe6e84e7816" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 41243 } } peers { permanent_uuid: "8278e8fef9c6406b84c17855fa85646d" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 41525 } } }
I20251026 08:51:09.812208 887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:09.813804 889 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:51:09.815383 889 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:51:09.825625 889 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: Generated new TSK 0
I20251026 08:51:09.826647 889 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:51:10.468675 640 catalog_manager.cc:1269] Loaded cluster ID: a0874ce9f7f14163992c32ce892c6494
I20251026 08:51:10.468781 640 catalog_manager.cc:1562] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: loading cluster ID for follower catalog manager: success
I20251026 08:51:10.470986 640 catalog_manager.cc:1584] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: acquiring CA information for follower catalog manager: success
I20251026 08:51:10.472146 640 catalog_manager.cc:1612] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 08:51:10.473958 651 catalog_manager.cc:1269] Loaded cluster ID: a0874ce9f7f14163992c32ce892c6494
I20251026 08:51:10.474048 651 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: loading cluster ID for follower catalog manager: success
I20251026 08:51:10.475567 651 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: acquiring CA information for follower catalog manager: success
I20251026 08:51:10.476375 651 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 08:51:10.581068 573 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35584
I20251026 08:51:10.663393 573 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35590
I20251026 08:51:10.737592 573 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:35604
I20251026 08:51:11.448372 623 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:11.582587 723 heartbeater.cc:499] Master 127.0.103.124:41525 was elected leader, sending a full tablet report...
I20251026 08:51:11.664429 801 heartbeater.cc:499] Master 127.0.103.124:41525 was elected leader, sending a full tablet report...
I20251026 08:51:11.738858 879 heartbeater.cc:499] Master 127.0.103.124:41525 was elected leader, sending a full tablet report...
I20251026 08:51:11.907656 413 test_util.cc:276] Using random seed: -495397078
I20251026 08:51:11.941587 573 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35606:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 08:51:11.984061 844 tablet_service.cc:1505] Processing CreateTablet for tablet 780c733af85e46f5b7a4fbb5aba997b1 (DEFAULT_TABLE table=test-workload [id=572ef26cd27c47b39cbf8931eab49580]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:51:11.984875 844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 780c733af85e46f5b7a4fbb5aba997b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:11.987123 688 tablet_service.cc:1505] Processing CreateTablet for tablet 978b3e45e32941c89a90281ce3169eed (DEFAULT_TABLE table=test-workload [id=572ef26cd27c47b39cbf8931eab49580]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:51:11.987354 687 tablet_service.cc:1505] Processing CreateTablet for tablet bd5f9f05984346feb21529a8e5f259f1 (DEFAULT_TABLE table=test-workload [id=572ef26cd27c47b39cbf8931eab49580]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:51:11.987715 688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 978b3e45e32941c89a90281ce3169eed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:11.987968 687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd5f9f05984346feb21529a8e5f259f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:11.990736 766 tablet_service.cc:1505] Processing CreateTablet for tablet e3cce1e4b57d48feb80a7c2caa86ca2e (DEFAULT_TABLE table=test-workload [id=572ef26cd27c47b39cbf8931eab49580]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:51:11.991151 766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cce1e4b57d48feb80a7c2caa86ca2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:11.997221 927 tablet_bootstrap.cc:492] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527: Bootstrap starting.
I20251026 08:51:12.000335 927 tablet_bootstrap.cc:654] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:12.000972 926 tablet_bootstrap.cc:492] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999: Bootstrap starting.
I20251026 08:51:12.001667 925 tablet_bootstrap.cc:492] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f: Bootstrap starting.
I20251026 08:51:12.002739 927 tablet_bootstrap.cc:492] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527: No bootstrap required, opened a new log
I20251026 08:51:12.002882 927 ts_tablet_manager.cc:1403] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 08:51:12.003468 925 tablet_bootstrap.cc:654] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:12.003468 926 tablet_bootstrap.cc:654] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:12.003535 927 raft_consensus.cc:359] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d49db780211440118377f5cb26102527" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35895 } }
I20251026 08:51:12.003700 927 raft_consensus.cc:385] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:12.004328 927 raft_consensus.cc:740] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49db780211440118377f5cb26102527, State: Initialized, Role: FOLLOWER
I20251026 08:51:12.004576 927 consensus_queue.cc:260] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d49db780211440118377f5cb26102527" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35895 } }
I20251026 08:51:12.004735 927 raft_consensus.cc:399] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:12.004830 927 raft_consensus.cc:493] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:12.004949 927 raft_consensus.cc:3060] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:12.006150 926 tablet_bootstrap.cc:492] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999: No bootstrap required, opened a new log
I20251026 08:51:12.006220 925 tablet_bootstrap.cc:492] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f: No bootstrap required, opened a new log
I20251026 08:51:12.006284 926 ts_tablet_manager.cc:1403] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:51:12.006325 925 ts_tablet_manager.cc:1403] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:51:12.007050 926 raft_consensus.cc:359] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.007086 925 raft_consensus.cc:359] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6337e065e933404b8bb90bc96193e77f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35335 } }
I20251026 08:51:12.007081 927 raft_consensus.cc:515] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d49db780211440118377f5cb26102527" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35895 } }
I20251026 08:51:12.007248 926 raft_consensus.cc:385] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:12.007272 925 raft_consensus.cc:385] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:12.007349 926 raft_consensus.cc:740] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725a688880b7412e87454cf66de08999, State: Initialized, Role: FOLLOWER
I20251026 08:51:12.007385 925 raft_consensus.cc:740] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6337e065e933404b8bb90bc96193e77f, State: Initialized, Role: FOLLOWER
I20251026 08:51:12.007411 927 leader_election.cc:304] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d49db780211440118377f5cb26102527; no voters:
I20251026 08:51:12.007591 926 consensus_queue.cc:260] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.007632 925 consensus_queue.cc:260] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6337e065e933404b8bb90bc96193e77f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35335 } }
I20251026 08:51:12.007791 926 raft_consensus.cc:399] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:12.007817 925 raft_consensus.cc:399] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:12.007948 926 raft_consensus.cc:493] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:12.007977 925 raft_consensus.cc:493] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:12.008224 927 leader_election.cc:290] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:12.008245 925 raft_consensus.cc:3060] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:12.008266 926 raft_consensus.cc:3060] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:12.008548 931 raft_consensus.cc:2804] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:12.011230 926 raft_consensus.cc:515] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.011231 925 raft_consensus.cc:515] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6337e065e933404b8bb90bc96193e77f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35335 } }
I20251026 08:51:12.011461 926 leader_election.cc:304] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 725a688880b7412e87454cf66de08999; no voters:
I20251026 08:51:12.011483 925 leader_election.cc:304] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6337e065e933404b8bb90bc96193e77f; no voters:
I20251026 08:51:12.015466 927 ts_tablet_manager.cc:1434] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
I20251026 08:51:12.015817 931 raft_consensus.cc:697] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 1 LEADER]: Becoming Leader. State: Replica: d49db780211440118377f5cb26102527, State: Running, Role: LEADER
I20251026 08:51:12.016131 931 consensus_queue.cc:237] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d49db780211440118377f5cb26102527" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35895 } }
I20251026 08:51:12.018126 935 raft_consensus.cc:2804] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:12.018528 925 leader_election.cc:290] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:12.018934 926 leader_election.cc:290] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:12.019066 934 raft_consensus.cc:2804] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:12.020907 573 catalog_manager.cc:5649] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 reported cstate change: term changed from 0 to 1, leader changed from <none> to d49db780211440118377f5cb26102527 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "d49db780211440118377f5cb26102527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d49db780211440118377f5cb26102527" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35895 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:12.026778 935 raft_consensus.cc:697] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 1 LEADER]: Becoming Leader. State: Replica: 6337e065e933404b8bb90bc96193e77f, State: Running, Role: LEADER
I20251026 08:51:12.029670 935 consensus_queue.cc:237] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6337e065e933404b8bb90bc96193e77f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35335 } }
I20251026 08:51:12.035486 934 raft_consensus.cc:697] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 1 LEADER]: Becoming Leader. State: Replica: 725a688880b7412e87454cf66de08999, State: Running, Role: LEADER
I20251026 08:51:12.035856 926 ts_tablet_manager.cc:1434] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999: Time spent starting tablet: real 0.029s user 0.011s sys 0.012s
I20251026 08:51:12.036604 926 tablet_bootstrap.cc:492] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999: Bootstrap starting.
I20251026 08:51:12.038538 934 consensus_queue.cc:237] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.038864 926 tablet_bootstrap.cc:654] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:12.039132 925 ts_tablet_manager.cc:1434] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f: Time spent starting tablet: real 0.030s user 0.012s sys 0.008s
I20251026 08:51:12.040913 572 catalog_manager.cc:5649] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6337e065e933404b8bb90bc96193e77f (127.0.103.67). New cstate: current_term: 1 leader_uuid: "6337e065e933404b8bb90bc96193e77f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6337e065e933404b8bb90bc96193e77f" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35335 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:12.046877 573 catalog_manager.cc:5649] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725a688880b7412e87454cf66de08999 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "725a688880b7412e87454cf66de08999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:12.090399 926 tablet_bootstrap.cc:492] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999: No bootstrap required, opened a new log
I20251026 08:51:12.090520 926 ts_tablet_manager.cc:1403] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.023s
I20251026 08:51:12.090965 926 raft_consensus.cc:359] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.091099 926 raft_consensus.cc:385] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:12.091150 926 raft_consensus.cc:740] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 725a688880b7412e87454cf66de08999, State: Initialized, Role: FOLLOWER
I20251026 08:51:12.091287 926 consensus_queue.cc:260] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.091375 926 raft_consensus.cc:399] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:12.091429 926 raft_consensus.cc:493] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:12.091485 926 raft_consensus.cc:3060] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:12.092955 926 raft_consensus.cc:515] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.093127 926 leader_election.cc:304] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 725a688880b7412e87454cf66de08999; no voters:
I20251026 08:51:12.093261 926 leader_election.cc:290] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:12.093322 934 raft_consensus.cc:2804] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:12.093515 934 raft_consensus.cc:697] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 1 LEADER]: Becoming Leader. State: Replica: 725a688880b7412e87454cf66de08999, State: Running, Role: LEADER
I20251026 08:51:12.093617 926 ts_tablet_manager.cc:1434] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20251026 08:51:12.093719 934 consensus_queue.cc:237] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } }
I20251026 08:51:12.095542 573 catalog_manager.cc:5649] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 reported cstate change: term changed from 0 to 1, leader changed from <none> to 725a688880b7412e87454cf66de08999 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "725a688880b7412e87454cf66de08999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "725a688880b7412e87454cf66de08999" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 36409 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:12.127908 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:51:12.137239 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:12.137759 413 tablet_replica.cc:333] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999: stopping tablet replica
I20251026 08:51:12.138137 413 raft_consensus.cc:2243] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:12.138363 413 raft_consensus.cc:2272] T 978b3e45e32941c89a90281ce3169eed P 725a688880b7412e87454cf66de08999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.139786 413 tablet_replica.cc:333] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999: stopping tablet replica
I20251026 08:51:12.139904 413 raft_consensus.cc:2243] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:12.140012 413 raft_consensus.cc:2272] T bd5f9f05984346feb21529a8e5f259f1 P 725a688880b7412e87454cf66de08999 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.144556 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:51:12.149467 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:51:12.157150 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:12.157455 413 tablet_replica.cc:333] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527: stopping tablet replica
I20251026 08:51:12.157622 413 raft_consensus.cc:2243] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:12.157768 413 raft_consensus.cc:2272] T e3cce1e4b57d48feb80a7c2caa86ca2e P d49db780211440118377f5cb26102527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.161445 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:51:12.163833 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:51:12.171792 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:12.172075 413 tablet_replica.cc:333] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f: stopping tablet replica
I20251026 08:51:12.172223 413 raft_consensus.cc:2243] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:12.172358 413 raft_consensus.cc:2272] T 780c733af85e46f5b7a4fbb5aba997b1 P 6337e065e933404b8bb90bc96193e77f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.175736 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:51:12.178109 413 master.cc:561] Master@127.0.103.126:33569 shutting down...
I20251026 08:51:12.185536 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:12.185691 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.185802 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 81d02c4294534063aef498aa325649a6: stopping tablet replica
I20251026 08:51:12.207950 413 master.cc:583] Master@127.0.103.126:33569 shutdown complete.
I20251026 08:51:12.211853 413 master.cc:561] Master@127.0.103.125:41243 shutting down...
I20251026 08:51:12.218436 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:12.218603 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.218703 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P db2b1a7b08c249a89892efe6e84e7816: stopping tablet replica
I20251026 08:51:12.232043 413 master.cc:583] Master@127.0.103.125:41243 shutdown complete.
I20251026 08:51:12.235101 413 master.cc:561] Master@127.0.103.124:41525 shutting down...
I20251026 08:51:12.241915 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:12.242306 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:12.242414 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8278e8fef9c6406b84c17855fa85646d: stopping tablet replica
I20251026 08:51:12.255541 413 master.cc:583] Master@127.0.103.124:41525 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (3036 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20251026 08:51:12.266947 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36713
I20251026 08:51:12.267431 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:12.269336 948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.269747 949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.269873 951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:12.270515 413 server_base.cc:1047] running on GCE node
I20251026 08:51:12.270738 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:12.270802 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:12.270849 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468672270834 us; error 0 us; skew 500 ppm
I20251026 08:51:12.271010 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:12.271961 413 webserver.cc:492] Webserver started at http://127.0.103.126:36161/ using document root <none> and password file <none>
I20251026 08:51:12.272166 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:12.272229 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:12.272377 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:12.272881 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "304fd43c9b094d91b7353751368098ec"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.275055 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:51:12.276301 956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.276598 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:12.276690 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "304fd43c9b094d91b7353751368098ec"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.276783 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:12.296598 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:12.297091 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:12.309101 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36713
I20251026 08:51:12.309123 1007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36713 every 8 connection(s)
I20251026 08:51:12.310179 1008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:12.313315 1008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec: Bootstrap starting.
I20251026 08:51:12.314438 1008 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:12.315655 1008 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec: No bootstrap required, opened a new log
I20251026 08:51:12.316146 1008 raft_consensus.cc:359] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304fd43c9b094d91b7353751368098ec" member_type: VOTER }
I20251026 08:51:12.316262 1008 raft_consensus.cc:385] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:12.316318 1008 raft_consensus.cc:740] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 304fd43c9b094d91b7353751368098ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:12.316497 1008 consensus_queue.cc:260] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304fd43c9b094d91b7353751368098ec" member_type: VOTER }
I20251026 08:51:12.316617 1008 raft_consensus.cc:399] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:12.316663 1008 raft_consensus.cc:493] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:12.316730 1008 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:12.318362 1008 raft_consensus.cc:515] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304fd43c9b094d91b7353751368098ec" member_type: VOTER }
I20251026 08:51:12.318521 1008 leader_election.cc:304] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 304fd43c9b094d91b7353751368098ec; no voters:
I20251026 08:51:12.318852 1008 leader_election.cc:290] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:12.319000 1011 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:12.319610 1011 raft_consensus.cc:697] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 1 LEADER]: Becoming Leader. State: Replica: 304fd43c9b094d91b7353751368098ec, State: Running, Role: LEADER
I20251026 08:51:12.319746 1008 sys_catalog.cc:565] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:12.319849 1011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304fd43c9b094d91b7353751368098ec" member_type: VOTER }
I20251026 08:51:12.320806 1013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 304fd43c9b094d91b7353751368098ec. Latest consensus state: current_term: 1 leader_uuid: "304fd43c9b094d91b7353751368098ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304fd43c9b094d91b7353751368098ec" member_type: VOTER } }
I20251026 08:51:12.320973 1013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:12.322224 1012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "304fd43c9b094d91b7353751368098ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "304fd43c9b094d91b7353751368098ec" member_type: VOTER } }
I20251026 08:51:12.322417 1012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:12.323203 1018 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:51:12.325382 1018 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:51:12.328024 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:51:12.328685 1018 catalog_manager.cc:1357] Generated new cluster ID: 0361d6e246044f6087c60884331f1597
I20251026 08:51:12.328758 1018 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:51:12.336567 1018 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:51:12.337200 1018 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:51:12.347018 1018 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec: Generated new TSK 0
I20251026 08:51:12.347203 1018 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:51:12.360693 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:12.363260 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.363425 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.364171 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:12.364426 413 server_base.cc:1047] running on GCE node
I20251026 08:51:12.364646 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:12.364706 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:12.364769 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468672364739 us; error 0 us; skew 500 ppm
I20251026 08:51:12.364898 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:12.365872 413 webserver.cc:492] Webserver started at http://127.0.103.65:34119/ using document root <none> and password file <none>
I20251026 08:51:12.366070 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:12.366187 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:12.366302 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:12.366807 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "70cb7be3748846a2a5f5083fb8e36da4"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.369114 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:51:12.370560 1037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.370834 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:12.370936 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "70cb7be3748846a2a5f5083fb8e36da4"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.371047 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:12.380445 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:12.380863 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:12.381297 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:12.381983 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:12.382064 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.382230 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:12.382282 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.395767 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:44655
I20251026 08:51:12.395792 1099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:44655 every 8 connection(s)
I20251026 08:51:12.397243 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:12.399811 1104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.400370 1105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.400635 1107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:12.403386 413 server_base.cc:1047] running on GCE node
I20251026 08:51:12.403601 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:12.403697 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:12.403759 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468672403733 us; error 0 us; skew 500 ppm
I20251026 08:51:12.403909 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:12.405084 413 webserver.cc:492] Webserver started at http://127.0.103.66:41725/ using document root <none> and password file <none>
I20251026 08:51:12.405140 1100 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36713
I20251026 08:51:12.405256 1100 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:12.405318 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:12.405391 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:12.405516 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:12.405598 1100 heartbeater.cc:507] Master 127.0.103.126:36713 requested a full tablet report, sending...
I20251026 08:51:12.406162 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.406433 973 ts_manager.cc:194] Registered new tserver with Master: 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655)
I20251026 08:51:12.407891 973 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33006
I20251026 08:51:12.409013 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20251026 08:51:12.410266 1112 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.410542 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:51:12.410637 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.410712 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:12.421317 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:12.421748 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:12.422219 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:12.422878 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:12.422943 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.423003 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:12.423050 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.436677 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:35057
I20251026 08:51:12.436729 1174 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:35057 every 8 connection(s)
I20251026 08:51:12.438580 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:12.440984 1178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.441195 1179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:12.442866 1181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:12.443034 413 server_base.cc:1047] running on GCE node
I20251026 08:51:12.443431 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:12.443511 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:12.443593 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468672443554 us; error 0 us; skew 500 ppm
I20251026 08:51:12.443737 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:12.444819 413 webserver.cc:492] Webserver started at http://127.0.103.67:33115/ using document root <none> and password file <none>
I20251026 08:51:12.445080 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:12.445194 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:12.445350 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:12.445991 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "146b811db9b842a29796d93521f78516"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.446785 1175 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36713
I20251026 08:51:12.446883 1175 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:12.447209 1175 heartbeater.cc:507] Master 127.0.103.126:36713 requested a full tablet report, sending...
I20251026 08:51:12.448019 973 ts_manager.cc:194] Registered new tserver with Master: fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057)
I20251026 08:51:12.448658 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251026 08:51:12.448949 973 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33020
I20251026 08:51:12.450655 1186 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.450969 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:12.451045 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "146b811db9b842a29796d93521f78516"
format_stamp: "Formatted at 2025-10-26 08:51:12 on dist-test-slave-rd83"
I20251026 08:51:12.451141 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:12.465052 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:12.465523 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:12.465936 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:12.466614 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:12.466667 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.466727 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:12.466763 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:12.481184 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:33163
I20251026 08:51:12.481218 1248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:33163 every 8 connection(s)
I20251026 08:51:12.487298 1249 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36713
I20251026 08:51:12.487422 1249 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:12.487772 1249 heartbeater.cc:507] Master 127.0.103.126:36713 requested a full tablet report, sending...
I20251026 08:51:12.488481 973 ts_manager.cc:194] Registered new tserver with Master: 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:12.488879 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006940776s
I20251026 08:51:12.489113 973 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33030
I20251026 08:51:13.409170 1100 heartbeater.cc:499] Master 127.0.103.126:36713 was elected leader, sending a full tablet report...
I20251026 08:51:13.450022 1175 heartbeater.cc:499] Master 127.0.103.126:36713 was elected leader, sending a full tablet report...
I20251026 08:51:13.490181 1249 heartbeater.cc:499] Master 127.0.103.126:36713 was elected leader, sending a full tablet report...
I20251026 08:51:13.515796 413 test_util.cc:276] Using random seed: -493788936
I20251026 08:51:13.526690 973 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33042:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:51:13.527464 973 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:51:13.560664 1213 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0536d698924dcfb5d376bc8089b17a (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251026 08:51:13.560724 1140 tablet_service.cc:1505] Processing CreateTablet for tablet ee699deac8b34f91849c02a7ff887bce (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251026 08:51:13.561213 1214 tablet_service.cc:1505] Processing CreateTablet for tablet ee699deac8b34f91849c02a7ff887bce (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251026 08:51:13.561923 1209 tablet_service.cc:1505] Processing CreateTablet for tablet 891cf73ca2ac4592aa22fa81a6a817f2 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251026 08:51:13.562546 1139 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0536d698924dcfb5d376bc8089b17a (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251026 08:51:13.562968 1207 tablet_service.cc:1505] Processing CreateTablet for tablet 7c8ae636b1dc4d2fbacb1c36993f04e3 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251026 08:51:13.561242 1213 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0536d698924dcfb5d376bc8089b17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.561309 1211 tablet_service.cc:1505] Processing CreateTablet for tablet a8a1daf9a2814a4a9ba917d0896ae4dc (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251026 08:51:13.564369 1208 tablet_service.cc:1505] Processing CreateTablet for tablet 3f2c155adb5c41bc896579a045c82065 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251026 08:51:13.566857 1139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0536d698924dcfb5d376bc8089b17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.566380 1138 tablet_service.cc:1505] Processing CreateTablet for tablet f94cd0189fb94eefb38bfb4a682ed642 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251026 08:51:13.567904 1211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a1daf9a2814a4a9ba917d0896ae4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.567368 1137 tablet_service.cc:1505] Processing CreateTablet for tablet a8a1daf9a2814a4a9ba917d0896ae4dc (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251026 08:51:13.569753 1210 tablet_service.cc:1505] Processing CreateTablet for tablet 8eea5573f15d4564aaef98006620de06 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251026 08:51:13.569216 1212 tablet_service.cc:1505] Processing CreateTablet for tablet f94cd0189fb94eefb38bfb4a682ed642 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251026 08:51:13.570294 1133 tablet_service.cc:1505] Processing CreateTablet for tablet 7c8ae636b1dc4d2fbacb1c36993f04e3 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251026 08:51:13.570569 1135 tablet_service.cc:1505] Processing CreateTablet for tablet 891cf73ca2ac4592aa22fa81a6a817f2 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251026 08:51:13.570963 1140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee699deac8b34f91849c02a7ff887bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.571058 1136 tablet_service.cc:1505] Processing CreateTablet for tablet 8eea5573f15d4564aaef98006620de06 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251026 08:51:13.571179 1133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8ae636b1dc4d2fbacb1c36993f04e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.571378 1208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f2c155adb5c41bc896579a045c82065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.571393 1134 tablet_service.cc:1505] Processing CreateTablet for tablet 3f2c155adb5c41bc896579a045c82065 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251026 08:51:13.572028 1135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 891cf73ca2ac4592aa22fa81a6a817f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.574132 1134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f2c155adb5c41bc896579a045c82065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.574673 1137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a1daf9a2814a4a9ba917d0896ae4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.574879 1136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eea5573f15d4564aaef98006620de06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.576790 1212 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f94cd0189fb94eefb38bfb4a682ed642. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.576838 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f94cd0189fb94eefb38bfb4a682ed642. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.581152 1209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 891cf73ca2ac4592aa22fa81a6a817f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.582233 1214 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee699deac8b34f91849c02a7ff887bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.582508 1210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eea5573f15d4564aaef98006620de06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.582741 1207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8ae636b1dc4d2fbacb1c36993f04e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.591440 1063 tablet_service.cc:1505] Processing CreateTablet for tablet f94cd0189fb94eefb38bfb4a682ed642 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20251026 08:51:13.591463 1064 tablet_service.cc:1505] Processing CreateTablet for tablet 0d0536d698924dcfb5d376bc8089b17a (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20251026 08:51:13.593688 1062 tablet_service.cc:1505] Processing CreateTablet for tablet a8a1daf9a2814a4a9ba917d0896ae4dc (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20251026 08:51:13.593748 1065 tablet_service.cc:1505] Processing CreateTablet for tablet ee699deac8b34f91849c02a7ff887bce (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION VALUES < 268435455
I20251026 08:51:13.596503 1060 tablet_service.cc:1505] Processing CreateTablet for tablet 891cf73ca2ac4592aa22fa81a6a817f2 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20251026 08:51:13.597182 1061 tablet_service.cc:1505] Processing CreateTablet for tablet 8eea5573f15d4564aaef98006620de06 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20251026 08:51:13.598567 1063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f94cd0189fb94eefb38bfb4a682ed642. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.597513 1059 tablet_service.cc:1505] Processing CreateTablet for tablet 3f2c155adb5c41bc896579a045c82065 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20251026 08:51:13.601089 1065 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee699deac8b34f91849c02a7ff887bce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.601332 1061 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8eea5573f15d4564aaef98006620de06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.601539 1060 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 891cf73ca2ac4592aa22fa81a6a817f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.601112 1058 tablet_service.cc:1505] Processing CreateTablet for tablet 7c8ae636b1dc4d2fbacb1c36993f04e3 (DEFAULT_TABLE table=test-workload [id=5805c5f3a99e447bb0e7d8e1ac2c7794]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20251026 08:51:13.602219 1064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d0536d698924dcfb5d376bc8089b17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.602437 1059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f2c155adb5c41bc896579a045c82065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.605759 1062 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a1daf9a2814a4a9ba917d0896ae4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.606526 1058 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c8ae636b1dc4d2fbacb1c36993f04e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:13.663041 1269 tablet_bootstrap.cc:492] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.665092 1271 tablet_bootstrap.cc:492] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.665856 1270 tablet_bootstrap.cc:492] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.667004 1269 tablet_bootstrap.cc:654] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.669096 1270 tablet_bootstrap.cc:654] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.669368 1269 tablet_bootstrap.cc:492] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.669494 1269 ts_tablet_manager.cc:1403] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20251026 08:51:13.670141 1269 raft_consensus.cc:359] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.670423 1269 raft_consensus.cc:385] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.670533 1269 raft_consensus.cc:740] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.670807 1270 tablet_bootstrap.cc:492] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.670782 1269 consensus_queue.cc:260] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.670933 1270 ts_tablet_manager.cc:1403] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20251026 08:51:13.671573 1271 tablet_bootstrap.cc:654] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.671552 1269 ts_tablet_manager.cc:1434] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:51:13.671819 1269 tablet_bootstrap.cc:492] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.672132 1270 raft_consensus.cc:359] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.672320 1270 raft_consensus.cc:385] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.672849 1270 raft_consensus.cc:740] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.673092 1270 consensus_queue.cc:260] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.674022 1269 tablet_bootstrap.cc:654] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.679919 1271 tablet_bootstrap.cc:492] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.680063 1271 ts_tablet_manager.cc:1403] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.007s
I20251026 08:51:13.680290 1269 tablet_bootstrap.cc:492] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.680488 1269 ts_tablet_manager.cc:1403] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20251026 08:51:13.680552 1270 ts_tablet_manager.cc:1434] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.010s user 0.002s sys 0.006s
I20251026 08:51:13.680809 1270 tablet_bootstrap.cc:492] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.680835 1271 raft_consensus.cc:359] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.681017 1271 raft_consensus.cc:385] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.681116 1271 raft_consensus.cc:740] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.681329 1269 raft_consensus.cc:359] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.681392 1271 consensus_queue.cc:260] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.681568 1269 raft_consensus.cc:385] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.681663 1269 raft_consensus.cc:740] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.681916 1269 consensus_queue.cc:260] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.682417 1270 tablet_bootstrap.cc:654] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.683578 1269 ts_tablet_manager.cc:1434] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 08:51:13.683941 1269 tablet_bootstrap.cc:492] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.686642 1269 tablet_bootstrap.cc:654] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.689929 1271 ts_tablet_manager.cc:1434] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20251026 08:51:13.690241 1271 tablet_bootstrap.cc:492] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.690454 1270 tablet_bootstrap.cc:492] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.690629 1270 ts_tablet_manager.cc:1403] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251026 08:51:13.691509 1270 raft_consensus.cc:359] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.691768 1270 raft_consensus.cc:385] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.691776 1271 tablet_bootstrap.cc:654] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.691877 1270 raft_consensus.cc:740] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.692217 1270 consensus_queue.cc:260] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.694904 1271 tablet_bootstrap.cc:492] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.694916 1270 ts_tablet_manager.cc:1434] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 08:51:13.695082 1271 ts_tablet_manager.cc:1403] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:51:13.695607 1270 tablet_bootstrap.cc:492] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.695726 1271 raft_consensus.cc:359] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.695937 1269 tablet_bootstrap.cc:492] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.695964 1271 raft_consensus.cc:385] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.696072 1271 raft_consensus.cc:740] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.696204 1269 ts_tablet_manager.cc:1403] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.001s
I20251026 08:51:13.696393 1271 consensus_queue.cc:260] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.697340 1271 ts_tablet_manager.cc:1434] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:51:13.697618 1270 tablet_bootstrap.cc:654] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.697657 1271 tablet_bootstrap.cc:492] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.697561 1269 raft_consensus.cc:359] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.697804 1269 raft_consensus.cc:385] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.697939 1269 raft_consensus.cc:740] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.698299 1269 consensus_queue.cc:260] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.699270 1269 ts_tablet_manager.cc:1434] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:51:13.699548 1269 tablet_bootstrap.cc:492] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.699525 1271 tablet_bootstrap.cc:654] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.701007 1269 tablet_bootstrap.cc:654] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.704532 1270 tablet_bootstrap.cc:492] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.704689 1270 ts_tablet_manager.cc:1403] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20251026 08:51:13.705386 1270 raft_consensus.cc:359] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.705542 1270 raft_consensus.cc:385] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.705642 1270 raft_consensus.cc:740] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.705770 1269 tablet_bootstrap.cc:492] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.705776 1271 tablet_bootstrap.cc:492] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.706393 1269 ts_tablet_manager.cc:1403] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 08:51:13.706399 1271 ts_tablet_manager.cc:1403] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.001s
I20251026 08:51:13.706336 1270 consensus_queue.cc:260] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.706835 1270 ts_tablet_manager.cc:1434] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 08:51:13.707085 1270 tablet_bootstrap.cc:492] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.707086 1269 raft_consensus.cc:359] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.707185 1271 raft_consensus.cc:359] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.707335 1269 raft_consensus.cc:385] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.707360 1271 raft_consensus.cc:385] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.707414 1269 raft_consensus.cc:740] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.707468 1271 raft_consensus.cc:740] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.707690 1269 consensus_queue.cc:260] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.707772 1271 consensus_queue.cc:260] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.708315 1271 ts_tablet_manager.cc:1434] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:51:13.708563 1271 tablet_bootstrap.cc:492] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.708735 1270 tablet_bootstrap.cc:654] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.708913 1269 ts_tablet_manager.cc:1434] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:51:13.709232 1269 tablet_bootstrap.cc:492] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.710151 1271 tablet_bootstrap.cc:654] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.711131 1269 tablet_bootstrap.cc:654] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.716905 1269 tablet_bootstrap.cc:492] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.717104 1269 ts_tablet_manager.cc:1403] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251026 08:51:13.718030 1269 raft_consensus.cc:359] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.718294 1269 raft_consensus.cc:385] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.718395 1269 raft_consensus.cc:740] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.718664 1269 consensus_queue.cc:260] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.720022 1269 ts_tablet_manager.cc:1434] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:51:13.720361 1269 tablet_bootstrap.cc:492] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.722592 1269 tablet_bootstrap.cc:654] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.723892 1270 tablet_bootstrap.cc:492] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.724038 1270 ts_tablet_manager.cc:1403] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.016s
I20251026 08:51:13.724697 1270 raft_consensus.cc:359] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.724853 1270 raft_consensus.cc:385] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.724933 1270 raft_consensus.cc:740] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.725224 1270 consensus_queue.cc:260] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.726094 1270 ts_tablet_manager.cc:1434] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:51:13.726099 1269 tablet_bootstrap.cc:492] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.726289 1269 ts_tablet_manager.cc:1403] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20251026 08:51:13.726377 1270 tablet_bootstrap.cc:492] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.726686 1271 tablet_bootstrap.cc:492] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.726824 1271 ts_tablet_manager.cc:1403] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.016s
I20251026 08:51:13.726990 1269 raft_consensus.cc:359] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.727196 1269 raft_consensus.cc:385] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.727283 1269 raft_consensus.cc:740] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.728021 1271 raft_consensus.cc:359] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.728019 1269 consensus_queue.cc:260] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.728240 1271 raft_consensus.cc:385] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.728325 1271 raft_consensus.cc:740] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.728529 1269 ts_tablet_manager.cc:1434] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:51:13.728587 1270 tablet_bootstrap.cc:654] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.728535 1271 consensus_queue.cc:260] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.728819 1269 tablet_bootstrap.cc:492] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.729540 1271 ts_tablet_manager.cc:1434] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 08:51:13.729815 1271 tablet_bootstrap.cc:492] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.730340 1269 tablet_bootstrap.cc:654] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.731410 1271 tablet_bootstrap.cc:654] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.736349 1270 tablet_bootstrap.cc:492] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.736501 1270 ts_tablet_manager.cc:1403] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20251026 08:51:13.737243 1270 raft_consensus.cc:359] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.737433 1270 raft_consensus.cc:385] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.737541 1270 raft_consensus.cc:740] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.737804 1270 consensus_queue.cc:260] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.739019 1270 ts_tablet_manager.cc:1434] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:51:13.739305 1270 tablet_bootstrap.cc:492] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.740060 1271 tablet_bootstrap.cc:492] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.740203 1271 ts_tablet_manager.cc:1403] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20251026 08:51:13.740286 1269 tablet_bootstrap.cc:492] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.740417 1269 ts_tablet_manager.cc:1403] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20251026 08:51:13.740765 1274 raft_consensus.cc:493] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.740908 1274 raft_consensus.cc:515] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.741662 1271 raft_consensus.cc:359] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.741883 1271 raft_consensus.cc:385] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.741968 1271 raft_consensus.cc:740] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.741727 1269 raft_consensus.cc:359] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.742190 1269 raft_consensus.cc:385] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.742296 1269 raft_consensus.cc:740] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.742533 1269 consensus_queue.cc:260] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.742614 1274 leader_election.cc:290] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.743039 1270 tablet_bootstrap.cc:654] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.742925 1271 consensus_queue.cc:260] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.743451 1271 ts_tablet_manager.cc:1434] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:51:13.743710 1271 tablet_bootstrap.cc:492] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.745781 1271 tablet_bootstrap.cc:654] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.757355 1274 raft_consensus.cc:493] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.757555 1274 raft_consensus.cc:515] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.757900 1269 ts_tablet_manager.cc:1434] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.017s user 0.003s sys 0.006s
I20251026 08:51:13.758287 1269 tablet_bootstrap.cc:492] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec: Bootstrap starting.
I20251026 08:51:13.758296 1274 leader_election.cc:290] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.759433 1270 tablet_bootstrap.cc:492] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.759593 1270 ts_tablet_manager.cc:1403] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20251026 08:51:13.760077 1269 tablet_bootstrap.cc:654] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.760330 1270 raft_consensus.cc:359] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.760556 1270 raft_consensus.cc:385] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.760637 1270 raft_consensus.cc:740] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.760972 1270 consensus_queue.cc:260] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.761610 1270 ts_tablet_manager.cc:1434] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:51:13.761948 1270 tablet_bootstrap.cc:492] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.764531 1270 tablet_bootstrap.cc:654] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.765199 1271 tablet_bootstrap.cc:492] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.765535 1271 ts_tablet_manager.cc:1403] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.007s
I20251026 08:51:13.767309 1271 raft_consensus.cc:359] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.767510 1271 raft_consensus.cc:385] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.767592 1271 raft_consensus.cc:740] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.767848 1271 consensus_queue.cc:260] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.768375 1271 ts_tablet_manager.cc:1434] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:51:13.768582 1271 tablet_bootstrap.cc:492] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.766999 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.769033 1075 raft_consensus.cc:2468] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.767222 1074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eea5573f15d4564aaef98006620de06" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.769531 1114 leader_election.cc:304] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.769778 1274 raft_consensus.cc:2804] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.769778 1074 raft_consensus.cc:2468] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.769927 1274 raft_consensus.cc:493] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.770087 1274 raft_consensus.cc:3060] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.770220 1271 tablet_bootstrap.cc:654] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.770270 1114 leader_election.cc:304] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.770525 1292 raft_consensus.cc:2804] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.770671 1292 raft_consensus.cc:493] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.771001 1292 raft_consensus.cc:3060] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.774602 1269 tablet_bootstrap.cc:492] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec: No bootstrap required, opened a new log
I20251026 08:51:13.774767 1270 tablet_bootstrap.cc:492] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.776172 1274 raft_consensus.cc:515] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.776592 1270 ts_tablet_manager.cc:1403] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20251026 08:51:13.777217 1274 leader_election.cc:290] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Requested vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.777328 1278 raft_consensus.cc:493] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.777787 1269 ts_tablet_manager.cc:1403] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20251026 08:51:13.776358 1224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.778801 1224 raft_consensus.cc:2468] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.778131 1223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eea5573f15d4564aaef98006620de06" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.779771 1074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:13.778831 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.780179 1074 raft_consensus.cc:3060] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.780256 1222 raft_consensus.cc:3060] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.780613 1270 raft_consensus.cc:359] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.780797 1270 raft_consensus.cc:385] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.780905 1270 raft_consensus.cc:740] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.781013 1269 raft_consensus.cc:359] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.781217 1269 raft_consensus.cc:385] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.781309 1269 raft_consensus.cc:740] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.781507 1269 consensus_queue.cc:260] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.781988 1269 ts_tablet_manager.cc:1434] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251026 08:51:13.781211 1292 raft_consensus.cc:515] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.781213 1270 consensus_queue.cc:260] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.782663 1270 ts_tablet_manager.cc:1434] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251026 08:51:13.782757 1292 leader_election.cc:290] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Requested vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.781992 1271 tablet_bootstrap.cc:492] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.782990 1270 tablet_bootstrap.cc:492] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516: Bootstrap starting.
I20251026 08:51:13.782999 1271 ts_tablet_manager.cc:1403] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.006s
I20251026 08:51:13.784373 1271 raft_consensus.cc:359] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.784569 1271 raft_consensus.cc:385] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.784686 1271 raft_consensus.cc:740] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.778352 1278 raft_consensus.cc:515] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.784894 1271 consensus_queue.cc:260] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.785666 1270 tablet_bootstrap.cc:654] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.786164 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eea5573f15d4564aaef98006620de06" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:13.786330 1075 raft_consensus.cc:3060] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.788622 1223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eea5573f15d4564aaef98006620de06" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
W20251026 08:51:13.789834 1114 leader_election.cc:343] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 146b811db9b842a29796d93521f78516 (127.0.103.67:33163): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:51:13.790143 1278 leader_election.cc:290] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
W20251026 08:51:13.796304 1114 leader_election.cc:343] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 146b811db9b842a29796d93521f78516 (127.0.103.67:33163): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:51:13.795749 1222 raft_consensus.cc:2468] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.797192 1276 raft_consensus.cc:493] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.797623 1074 raft_consensus.cc:2468] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.799314 1114 leader_election.cc:304] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.800020 1292 raft_consensus.cc:2804] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.800208 1271 ts_tablet_manager.cc:1434] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.014s user 0.001s sys 0.010s
I20251026 08:51:13.797994 1276 raft_consensus.cc:515] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.801218 1292 raft_consensus.cc:697] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Leader. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:13.801674 1271 tablet_bootstrap.cc:492] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4: Bootstrap starting.
I20251026 08:51:13.802107 1292 consensus_queue.cc:237] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.808887 1276 leader_election.cc:290] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057)
I20251026 08:51:13.809624 1075 raft_consensus.cc:2468] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.813513 1114 leader_election.cc:304] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters: 146b811db9b842a29796d93521f78516
I20251026 08:51:13.816890 1274 raft_consensus.cc:2804] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.830710 1307 raft_consensus.cc:493] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.832314 1270 tablet_bootstrap.cc:492] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516: No bootstrap required, opened a new log
I20251026 08:51:13.832494 1270 ts_tablet_manager.cc:1403] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.022s
I20251026 08:51:13.833371 1270 raft_consensus.cc:359] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.833559 1270 raft_consensus.cc:385] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.833657 1270 raft_consensus.cc:740] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146b811db9b842a29796d93521f78516, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.833914 1270 consensus_queue.cc:260] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.835047 973 catalog_manager.cc:5649] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec reported cstate change: term changed from 0 to 1, leader changed from <none> to fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66). New cstate: current_term: 1 leader_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.836746 1270 ts_tablet_manager.cc:1434] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251026 08:51:13.840546 1274 raft_consensus.cc:697] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Leader. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:13.831224 1307 raft_consensus.cc:515] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.841818 1271 tablet_bootstrap.cc:654] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:13.843665 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.843711 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.843201 1274 consensus_queue.cc:237] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.844334 1222 raft_consensus.cc:2468] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.844820 1075 raft_consensus.cc:2468] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.845207 1307 leader_election.cc:290] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.845831 1114 leader_election.cc:304] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.848666 1307 raft_consensus.cc:2804] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.848848 1307 raft_consensus.cc:493] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.849007 1307 raft_consensus.cc:3060] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.850029 1271 tablet_bootstrap.cc:492] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4: No bootstrap required, opened a new log
I20251026 08:51:13.851577 1271 ts_tablet_manager.cc:1403] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.007s
I20251026 08:51:13.851886 973 catalog_manager.cc:5649] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec reported cstate change: term changed from 0 to 1, leader changed from <none> to fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66). New cstate: current_term: 1 leader_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.852880 1271 raft_consensus.cc:359] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.853191 1271 raft_consensus.cc:385] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:13.853295 1271 raft_consensus.cc:740] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Initialized, Role: FOLLOWER
I20251026 08:51:13.854077 1271 consensus_queue.cc:260] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.854563 1271 ts_tablet_manager.cc:1434] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251026 08:51:13.856096 1307 raft_consensus.cc:515] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.857508 1307 leader_election.cc:290] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Requested vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.857513 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:13.857918 1075 raft_consensus.cc:3060] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.857970 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.858301 1222 raft_consensus.cc:3060] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.862269 1222 raft_consensus.cc:2468] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.862660 1114 leader_election.cc:304] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.863476 1075 raft_consensus.cc:2468] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.862718 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a1daf9a2814a4a9ba917d0896ae4dc" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" is_pre_election: true
I20251026 08:51:13.864189 1307 raft_consensus.cc:2804] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.864233 1150 raft_consensus.cc:2468] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 0.
I20251026 08:51:13.864089 1274 raft_consensus.cc:493] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.865247 1039 leader_election.cc:304] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.865216 1274 raft_consensus.cc:515] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.865700 1300 raft_consensus.cc:2804] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.865808 1300 raft_consensus.cc:493] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.865913 1307 raft_consensus.cc:697] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Leader. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:13.866592 1300 raft_consensus.cc:3060] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.866550 1276 raft_consensus.cc:493] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.867049 1276 raft_consensus.cc:515] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.866803 1307 consensus_queue.cc:237] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.868126 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee699deac8b34f91849c02a7ff887bce" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.868677 1276 leader_election.cc:290] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057)
I20251026 08:51:13.868804 1274 leader_election.cc:290] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 146b811db9b842a29796d93521f78516 (127.0.103.67:33163), 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655)
I20251026 08:51:13.868937 1222 raft_consensus.cc:2468] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.868939 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee699deac8b34f91849c02a7ff887bce" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.869522 1075 raft_consensus.cc:2468] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.869531 1114 leader_election.cc:304] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.870152 1274 raft_consensus.cc:2804] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.870280 1274 raft_consensus.cc:493] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.870378 1274 raft_consensus.cc:3060] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.870555 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a1daf9a2814a4a9ba917d0896ae4dc" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.870748 1222 raft_consensus.cc:2468] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 0.
I20251026 08:51:13.872076 1300 raft_consensus.cc:515] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.872509 1278 raft_consensus.cc:493] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.872802 1278 raft_consensus.cc:515] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.873854 1278 leader_election.cc:290] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.874567 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a1daf9a2814a4a9ba917d0896ae4dc" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
I20251026 08:51:13.874774 1150 raft_consensus.cc:3060] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.875353 1274 raft_consensus.cc:515] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.876428 1224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee699deac8b34f91849c02a7ff887bce" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.876641 1300 leader_election.cc:290] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 election: Requested vote from peers fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.877254 1274 leader_election.cc:290] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Requested vote from peers 146b811db9b842a29796d93521f78516 (127.0.103.67:33163), 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655)
I20251026 08:51:13.878216 1224 raft_consensus.cc:3060] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.879238 1073 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eea5573f15d4564aaef98006620de06" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.879637 1073 raft_consensus.cc:2393] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 146b811db9b842a29796d93521f78516 in current term 1: Already voted for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in this term.
I20251026 08:51:13.880084 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a1daf9a2814a4a9ba917d0896ae4dc" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.880237 1222 raft_consensus.cc:3060] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.880832 1224 raft_consensus.cc:2468] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.881031 1223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f94cd0189fb94eefb38bfb4a682ed642" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.878252 1149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f94cd0189fb94eefb38bfb4a682ed642" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" is_pre_election: true
I20251026 08:51:13.881489 1149 raft_consensus.cc:2468] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 0.
I20251026 08:51:13.881734 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ee699deac8b34f91849c02a7ff887bce" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:13.881814 1074 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a1daf9a2814a4a9ba917d0896ae4dc" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.881901 1075 raft_consensus.cc:3060] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.882270 1223 raft_consensus.cc:2468] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 0.
I20251026 08:51:13.882302 1114 leader_election.cc:304] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.882546 1150 raft_consensus.cc:2468] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 1.
I20251026 08:51:13.882622 1039 leader_election.cc:304] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, 70cb7be3748846a2a5f5083fb8e36da4; no voters:
I20251026 08:51:13.882843 1300 raft_consensus.cc:2804] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.882277 1074 raft_consensus.cc:2393] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 146b811db9b842a29796d93521f78516 in current term 1: Already voted for candidate 70cb7be3748846a2a5f5083fb8e36da4 in this term.
I20251026 08:51:13.882761 973 catalog_manager.cc:5649] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec reported cstate change: term changed from 0 to 1, leader changed from <none> to fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66). New cstate: current_term: 1 leader_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.882967 1300 raft_consensus.cc:493] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.883046 1300 raft_consensus.cc:3060] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.883472 1039 leader_election.cc:304] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.883726 1278 raft_consensus.cc:2804] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.883937 1222 raft_consensus.cc:2468] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 1.
I20251026 08:51:13.885654 1300 raft_consensus.cc:515] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.885903 1075 raft_consensus.cc:2468] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.886263 1300 leader_election.cc:290] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 election: Requested vote from peers fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.887037 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f94cd0189fb94eefb38bfb4a682ed642" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
I20251026 08:51:13.887262 1150 raft_consensus.cc:3060] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.887369 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f94cd0189fb94eefb38bfb4a682ed642" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.888180 1274 raft_consensus.cc:2804] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.888387 1274 raft_consensus.cc:697] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Leader. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:13.888015 1222 raft_consensus.cc:3060] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.888741 1274 consensus_queue.cc:237] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } }
I20251026 08:51:13.890443 1150 raft_consensus.cc:2468] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 1.
I20251026 08:51:13.890749 1039 leader_election.cc:304] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.890997 1300 raft_consensus.cc:2804] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.891170 1300 raft_consensus.cc:697] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Becoming Leader. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Running, Role: LEADER
I20251026 08:51:13.891208 1222 raft_consensus.cc:2468] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 1.
I20251026 08:51:13.891445 1300 consensus_queue.cc:237] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.893234 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a8a1daf9a2814a4a9ba917d0896ae4dc" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" is_pre_election: true
I20251026 08:51:13.893273 1149 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8eea5573f15d4564aaef98006620de06" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" is_pre_election: true
I20251026 08:51:13.893481 1150 raft_consensus.cc:2393] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 146b811db9b842a29796d93521f78516 in current term 1: Already voted for candidate 70cb7be3748846a2a5f5083fb8e36da4 in this term.
I20251026 08:51:13.894016 1188 leader_election.cc:304] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 146b811db9b842a29796d93521f78516; no voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec
I20251026 08:51:13.894392 1276 raft_consensus.cc:2749] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 08:51:13.894872 1188 leader_election.cc:304] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 146b811db9b842a29796d93521f78516; no voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec
I20251026 08:51:13.895134 1276 raft_consensus.cc:3060] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.897097 1278 raft_consensus.cc:697] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Becoming Leader. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Running, Role: LEADER
I20251026 08:51:13.897483 1278 consensus_queue.cc:237] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.899442 1276 raft_consensus.cc:2749] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 08:51:13.901670 973 catalog_manager.cc:5649] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec reported cstate change: term changed from 0 to 1, leader changed from <none> to fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66). New cstate: current_term: 1 leader_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.901805 972 catalog_manager.cc:5649] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "70cb7be3748846a2a5f5083fb8e36da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.905932 972 catalog_manager.cc:5649] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "70cb7be3748846a2a5f5083fb8e36da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.934275 1274 raft_consensus.cc:493] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.934445 1274 raft_consensus.cc:515] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.934873 1274 leader_election.cc:290] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.935263 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8ae636b1dc4d2fbacb1c36993f04e3" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4" is_pre_election: true
I20251026 08:51:13.935269 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8ae636b1dc4d2fbacb1c36993f04e3" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.935508 1222 raft_consensus.cc:2468] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.935527 1075 raft_consensus.cc:2468] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 0.
I20251026 08:51:13.935855 1114 leader_election.cc:304] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.936085 1274 raft_consensus.cc:2804] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.936179 1274 raft_consensus.cc:493] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.936281 1274 raft_consensus.cc:3060] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.938014 1274 raft_consensus.cc:515] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.938540 1274 leader_election.cc:290] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Requested vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.938874 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8ae636b1dc4d2fbacb1c36993f04e3" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:13.938916 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c8ae636b1dc4d2fbacb1c36993f04e3" candidate_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.939059 1222 raft_consensus.cc:3060] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.939065 1075 raft_consensus.cc:3060] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.939963 1300 raft_consensus.cc:493] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:13.940137 1300 raft_consensus.cc:515] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.940678 1300 leader_election.cc:290] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.941064 1223 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d0536d698924dcfb5d376bc8089b17a" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516" is_pre_election: true
I20251026 08:51:13.941231 1223 raft_consensus.cc:2468] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 0.
I20251026 08:51:13.941269 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d0536d698924dcfb5d376bc8089b17a" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" is_pre_election: true
I20251026 08:51:13.941402 1150 raft_consensus.cc:2468] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 0.
I20251026 08:51:13.941753 1075 raft_consensus.cc:2468] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.941754 1222 raft_consensus.cc:2468] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe20ee26d64d4ca6827c9dd40d3c82ec in term 1.
I20251026 08:51:13.941787 1039 leader_election.cc:304] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, 70cb7be3748846a2a5f5083fb8e36da4; no voters:
I20251026 08:51:13.942112 1300 raft_consensus.cc:2804] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:13.942139 1114 leader_election.cc:304] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70cb7be3748846a2a5f5083fb8e36da4, fe20ee26d64d4ca6827c9dd40d3c82ec; no voters:
I20251026 08:51:13.942214 1300 raft_consensus.cc:493] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:13.942319 1300 raft_consensus.cc:3060] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.942375 1274 raft_consensus.cc:2804] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.942546 1274 raft_consensus.cc:697] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Leader. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:13.942822 1274 consensus_queue.cc:237] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.944687 1300 raft_consensus.cc:515] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.945186 1300 leader_election.cc:290] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 election: Requested vote from peers fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057), 146b811db9b842a29796d93521f78516 (127.0.103.67:33163)
I20251026 08:51:13.945222 972 catalog_manager.cc:5649] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec reported cstate change: term changed from 0 to 1, leader changed from <none> to fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66). New cstate: current_term: 1 leader_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:13.945499 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d0536d698924dcfb5d376bc8089b17a" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
I20251026 08:51:13.945585 1222 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d0536d698924dcfb5d376bc8089b17a" candidate_uuid: "70cb7be3748846a2a5f5083fb8e36da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146b811db9b842a29796d93521f78516"
I20251026 08:51:13.945654 1150 raft_consensus.cc:3060] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.946249 1222 raft_consensus.cc:3060] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:13.948745 1150 raft_consensus.cc:2468] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 1.
I20251026 08:51:13.948853 1222 raft_consensus.cc:2468] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70cb7be3748846a2a5f5083fb8e36da4 in term 1.
I20251026 08:51:13.949191 1039 leader_election.cc:304] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, 70cb7be3748846a2a5f5083fb8e36da4; no voters:
I20251026 08:51:13.949736 1300 raft_consensus.cc:2804] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:13.949887 1300 raft_consensus.cc:697] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Becoming Leader. State: Replica: 70cb7be3748846a2a5f5083fb8e36da4, State: Running, Role: LEADER
I20251026 08:51:13.950120 1300 consensus_queue.cc:237] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:13.951958 972 catalog_manager.cc:5649] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "70cb7be3748846a2a5f5083fb8e36da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:14.020167 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:14.023475 1320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:14.023499 1321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:14.023976 1323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:14.024659 413 server_base.cc:1047] running on GCE node
I20251026 08:51:14.024909 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:14.024983 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:14.025058 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468674025039 us; error 0 us; skew 500 ppm
I20251026 08:51:14.025249 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:14.026438 413 webserver.cc:492] Webserver started at http://127.0.103.68:34161/ using document root <none> and password file <none>
I20251026 08:51:14.026661 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:14.026736 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:14.027379 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:14.027990 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "b7f096f2a8894c43a4027a38fd280053"
format_stamp: "Formatted at 2025-10-26 08:51:14 on dist-test-slave-rd83"
I20251026 08:51:14.030313 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20251026 08:51:14.031699 1328 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:14.032063 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:14.032184 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "b7f096f2a8894c43a4027a38fd280053"
format_stamp: "Formatted at 2025-10-26 08:51:14 on dist-test-slave-rd83"
I20251026 08:51:14.032301 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:14.039806 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:14.040385 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:14.040921 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:14.041815 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:14.041894 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:14.042001 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:14.042120 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:14.062361 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:34509
I20251026 08:51:14.062381 1390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:34509 every 8 connection(s)
I20251026 08:51:14.069278 1391 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36713
I20251026 08:51:14.069399 1391 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:14.070207 1391 heartbeater.cc:507] Master 127.0.103.126:36713 requested a full tablet report, sending...
I20251026 08:51:14.070981 972 ts_manager.cc:194] Registered new tserver with Master: b7f096f2a8894c43a4027a38fd280053 (127.0.103.68:34509)
I20251026 08:51:14.071745 972 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33056
I20251026 08:51:14.289546 1274 consensus_queue.cc:1048] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.293506 1306 consensus_queue.cc:1048] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.299624 1307 consensus_queue.cc:1048] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.303157 1274 consensus_queue.cc:1048] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.305970 1274 consensus_queue.cc:1048] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.309370 1307 consensus_queue.cc:1048] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.326654 1300 consensus_queue.cc:1048] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.328194 1026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:14.331218 1313 consensus_queue.cc:1048] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.338444 1150 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065"
dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
mode: GRACEFUL
new_leader_uuid: "146b811db9b842a29796d93521f78516"
from {username='slave'} at 127.0.0.1:33650
I20251026 08:51:14.338651 1150 raft_consensus.cc:606] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Received request to transfer leadership to TS 146b811db9b842a29796d93521f78516
I20251026 08:51:14.339530 1150 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2"
dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
mode: GRACEFUL
new_leader_uuid: "146b811db9b842a29796d93521f78516"
from {username='slave'} at 127.0.0.1:33650
I20251026 08:51:14.339720 1150 raft_consensus.cc:606] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Received request to transfer leadership to TS 146b811db9b842a29796d93521f78516
I20251026 08:51:14.340023 1026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 08:51:14.340247 1026 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:14.361851 1313 consensus_queue.cc:1048] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.365695 1300 consensus_queue.cc:1048] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.394510 1292 consensus_queue.cc:1048] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.395306 1292 consensus_queue.cc:1048] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.397184 1274 raft_consensus.cc:993] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: : Instructing follower 146b811db9b842a29796d93521f78516 to start an election
I20251026 08:51:14.397367 1274 raft_consensus.cc:1081] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Signalling peer 146b811db9b842a29796d93521f78516 to start an election
I20251026 08:51:14.398677 1223 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065"
dest_uuid: "146b811db9b842a29796d93521f78516"
from {username='slave'} at 127.0.0.1:55416
I20251026 08:51:14.398851 1223 raft_consensus.cc:493] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:51:14.398994 1223 raft_consensus.cc:3060] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:14.401690 1223 raft_consensus.cc:515] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:14.402364 1223 leader_election.cc:290] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 2 election: Requested vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057)
I20251026 08:51:14.402616 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:14.402774 1075 raft_consensus.cc:3060] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:14.402804 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3f2c155adb5c41bc896579a045c82065" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
I20251026 08:51:14.402949 1150 raft_consensus.cc:3055] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:51:14.403034 1150 raft_consensus.cc:740] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:14.403317 1150 consensus_queue.cc:260] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:14.403637 1150 raft_consensus.cc:3060] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:14.405225 1075 raft_consensus.cc:2468] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146b811db9b842a29796d93521f78516 in term 2.
I20251026 08:51:14.405681 1188 leader_election.cc:304] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, 70cb7be3748846a2a5f5083fb8e36da4; no voters:
I20251026 08:51:14.405934 1276 raft_consensus.cc:2804] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:51:14.406024 1150 raft_consensus.cc:2468] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146b811db9b842a29796d93521f78516 in term 2.
I20251026 08:51:14.406347 1276 raft_consensus.cc:697] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 2 LEADER]: Becoming Leader. State: Replica: 146b811db9b842a29796d93521f78516, State: Running, Role: LEADER
I20251026 08:51:14.406569 1276 consensus_queue.cc:237] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:14.408293 973 catalog_manager.cc:5649] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 reported cstate change: term changed from 1 to 2, leader changed from fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66) to 146b811db9b842a29796d93521f78516 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "146b811db9b842a29796d93521f78516" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:14.414572 1307 consensus_queue.cc:1048] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.414757 1300 consensus_queue.cc:1048] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.421870 1278 consensus_queue.cc:1048] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.421998 1306 consensus_queue.cc:1048] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:14.692800 1306 raft_consensus.cc:993] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: : Instructing follower 146b811db9b842a29796d93521f78516 to start an election
I20251026 08:51:14.692951 1306 raft_consensus.cc:1081] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Signalling peer 146b811db9b842a29796d93521f78516 to start an election
I20251026 08:51:14.693501 1223 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2"
dest_uuid: "146b811db9b842a29796d93521f78516"
from {username='slave'} at 127.0.0.1:55416
I20251026 08:51:14.693687 1223 raft_consensus.cc:493] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:51:14.693792 1223 raft_consensus.cc:3060] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:14.695417 1223 raft_consensus.cc:515] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:14.695861 1223 leader_election.cc:290] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 2 election: Requested vote from peers 70cb7be3748846a2a5f5083fb8e36da4 (127.0.103.65:44655), fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66:35057)
I20251026 08:51:14.696185 1075 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "70cb7be3748846a2a5f5083fb8e36da4"
I20251026 08:51:14.696338 1075 raft_consensus.cc:3060] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:14.696316 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "891cf73ca2ac4592aa22fa81a6a817f2" candidate_uuid: "146b811db9b842a29796d93521f78516" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec"
I20251026 08:51:14.696518 1150 raft_consensus.cc:3055] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:51:14.696584 1150 raft_consensus.cc:740] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe20ee26d64d4ca6827c9dd40d3c82ec, State: Running, Role: LEADER
I20251026 08:51:14.696795 1150 consensus_queue.cc:260] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:14.697036 1150 raft_consensus.cc:3060] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:14.698395 1075 raft_consensus.cc:2468] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146b811db9b842a29796d93521f78516 in term 2.
I20251026 08:51:14.698748 1188 leader_election.cc:304] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146b811db9b842a29796d93521f78516, 70cb7be3748846a2a5f5083fb8e36da4; no voters:
I20251026 08:51:14.698983 1276 raft_consensus.cc:2804] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:51:14.699003 1150 raft_consensus.cc:2468] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 146b811db9b842a29796d93521f78516 in term 2.
I20251026 08:51:14.699101 1276 raft_consensus.cc:697] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 2 LEADER]: Becoming Leader. State: Replica: 146b811db9b842a29796d93521f78516, State: Running, Role: LEADER
I20251026 08:51:14.699385 1276 consensus_queue.cc:237] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } }
I20251026 08:51:14.700994 972 catalog_manager.cc:5649] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 reported cstate change: term changed from 1 to 2, leader changed from fe20ee26d64d4ca6827c9dd40d3c82ec (127.0.103.66) to 146b811db9b842a29796d93521f78516 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "146b811db9b842a29796d93521f78516" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:14.783635 1075 raft_consensus.cc:1275] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Refusing update from remote peer 146b811db9b842a29796d93521f78516: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:51:14.784034 1276 consensus_queue.cc:1048] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:14.786679 1150 raft_consensus.cc:1275] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Refusing update from remote peer 146b811db9b842a29796d93521f78516: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:51:14.787490 1416 consensus_queue.cc:1048] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:15.072877 1391 heartbeater.cc:499] Master 127.0.103.126:36713 was elected leader, sending a full tablet report...
I20251026 08:51:15.122879 1150 raft_consensus.cc:1275] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Refusing update from remote peer 146b811db9b842a29796d93521f78516: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:51:15.123345 1416 consensus_queue.cc:1048] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:15.125793 1075 raft_consensus.cc:1275] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Refusing update from remote peer 146b811db9b842a29796d93521f78516: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:51:15.126287 1416 consensus_queue.cc:1048] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:16.340541 1026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:16.341642 1026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:16.341789 1026 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:18.342156 1026 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:18.343302 1026 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:18.343470 1026 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:19.338812 1074 consensus_queue.cc:237] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } attrs { replace: true } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } }
I20251026 08:51:19.341001 1150 raft_consensus.cc:1275] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Refusing update from remote peer 70cb7be3748846a2a5f5083fb8e36da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:19.341907 1223 raft_consensus.cc:1275] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Refusing update from remote peer 70cb7be3748846a2a5f5083fb8e36da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:19.341840 1421 consensus_queue.cc:1048] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:19.342439 1421 consensus_queue.cc:1048] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:19.344532 1313 raft_consensus.cc:2955] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } attrs { replace: true } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.345024 1150 raft_consensus.cc:2955] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } attrs { replace: true } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
W20251026 08:51:19.349792 1041 consensus_peers.cc:597] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 -> Peer b7f096f2a8894c43a4027a38fd280053 (127.0.103.68:34509): Couldn't send request to peer b7f096f2a8894c43a4027a38fd280053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a8a1daf9a2814a4a9ba917d0896ae4dc. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:51:19.350375 972 catalog_manager.cc:5649] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec reported cstate change: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "70cb7be3748846a2a5f5083fb8e36da4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } attrs { replace: true } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.350816 1222 raft_consensus.cc:2955] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } attrs { replace: true } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.359714 1223 consensus_queue.cc:237] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } }
I20251026 08:51:19.362133 1074 raft_consensus.cc:1275] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Refusing update from remote peer 146b811db9b842a29796d93521f78516: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:51:19.362761 1432 consensus_queue.cc:1048] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:51:19.363087 1150 raft_consensus.cc:1275] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Refusing update from remote peer 146b811db9b842a29796d93521f78516: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:51:19.363801 1432 consensus_queue.cc:1048] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:51:19.365736 1417 raft_consensus.cc:2955] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.366338 1074 raft_consensus.cc:2955] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.370052 1150 raft_consensus.cc:2955] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.370916 973 catalog_manager.cc:5649] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 reported cstate change: config changed from index -1 to 3, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "146b811db9b842a29796d93521f78516" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 08:51:19.374404 1190 consensus_peers.cc:597] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 -> Peer b7f096f2a8894c43a4027a38fd280053 (127.0.103.68:34509): Couldn't send request to peer b7f096f2a8894c43a4027a38fd280053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 891cf73ca2ac4592aa22fa81a6a817f2. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:51:19.381557 1150 consensus_queue.cc:237] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } }
I20251026 08:51:19.384264 1074 raft_consensus.cc:1275] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Refusing update from remote peer fe20ee26d64d4ca6827c9dd40d3c82ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:19.384418 1223 raft_consensus.cc:1275] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Refusing update from remote peer fe20ee26d64d4ca6827c9dd40d3c82ec: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:19.384879 1274 consensus_queue.cc:1048] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:19.385145 1439 consensus_queue.cc:1048] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:19.388926 1074 raft_consensus.cc:2955] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.390251 1223 raft_consensus.cc:2955] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
W20251026 08:51:19.391741 1116 consensus_peers.cc:597] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec -> Peer b7f096f2a8894c43a4027a38fd280053 (127.0.103.68:34509): Couldn't send request to peer b7f096f2a8894c43a4027a38fd280053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c8ae636b1dc4d2fbacb1c36993f04e3. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:51:19.388401 1439 raft_consensus.cc:2955] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.393551 972 catalog_manager.cc:5649] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 reported cstate change: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } attrs { replace: true } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.397886 1074 consensus_queue.cc:237] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } }
I20251026 08:51:19.401163 1223 raft_consensus.cc:1275] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Refusing update from remote peer 70cb7be3748846a2a5f5083fb8e36da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:19.401188 1150 raft_consensus.cc:1275] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Refusing update from remote peer 70cb7be3748846a2a5f5083fb8e36da4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 08:51:19.401517 1041 consensus_peers.cc:597] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 -> Peer b7f096f2a8894c43a4027a38fd280053 (127.0.103.68:34509): Couldn't send request to peer b7f096f2a8894c43a4027a38fd280053. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d0536d698924dcfb5d376bc8089b17a. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:51:19.401770 1444 consensus_queue.cc:1048] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:19.402083 1421 consensus_queue.cc:1048] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:19.404781 1313 raft_consensus.cc:2955] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.405282 1223 raft_consensus.cc:2955] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.405279 1150 raft_consensus.cc:2955] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.408859 973 catalog_manager.cc:5649] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 reported cstate change: config changed from index -1 to 2, NON_VOTER b7f096f2a8894c43a4027a38fd280053 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "70cb7be3748846a2a5f5083fb8e36da4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "70cb7be3748846a2a5f5083fb8e36da4" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44655 } } peers { permanent_uuid: "fe20ee26d64d4ca6827c9dd40d3c82ec" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 35057 } attrs { replace: true } } peers { permanent_uuid: "146b811db9b842a29796d93521f78516" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33163 } } peers { permanent_uuid: "b7f096f2a8894c43a4027a38fd280053" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34509 } attrs { promote: true } } }
I20251026 08:51:19.576936 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
W20251026 08:51:19.582558 958 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20251026 08:51:19.584687 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20251026 08:51:19.584811 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:51:19.584932 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:51:19.585047 413 tablet_replica.cc:333] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.585218 413 raft_consensus.cc:2243] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.585408 413 raft_consensus.cc:2272] T ee699deac8b34f91849c02a7ff887bce P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.585892 413 tablet_replica.cc:333] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.586053 413 raft_consensus.cc:2243] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.586242 413 raft_consensus.cc:2272] T 3f2c155adb5c41bc896579a045c82065 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251026 08:51:19.586318 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:51:19.586400 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:51:19.586812 413 tablet_replica.cc:333] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.586962 413 raft_consensus.cc:2243] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.587262 413 raft_consensus.cc:2272] T f94cd0189fb94eefb38bfb4a682ed642 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.587652 413 tablet_replica.cc:333] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.587834 413 raft_consensus.cc:2243] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.588032 413 raft_consensus.cc:2272] T 8eea5573f15d4564aaef98006620de06 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.588493 413 tablet_replica.cc:333] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.588618 413 raft_consensus.cc:2243] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.589016 413 raft_consensus.cc:2272] T a8a1daf9a2814a4a9ba917d0896ae4dc P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.589766 413 tablet_replica.cc:333] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.589938 413 raft_consensus.cc:2243] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.590164 413 raft_consensus.cc:2272] T 891cf73ca2ac4592aa22fa81a6a817f2 P 70cb7be3748846a2a5f5083fb8e36da4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.591102 413 tablet_replica.cc:333] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.591305 413 raft_consensus.cc:2243] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.591732 413 raft_consensus.cc:2272] T 0d0536d698924dcfb5d376bc8089b17a P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.592535 413 tablet_replica.cc:333] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4: stopping tablet replica
I20251026 08:51:19.592695 413 raft_consensus.cc:2243] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.592857 413 raft_consensus.cc:2272] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 70cb7be3748846a2a5f5083fb8e36da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.599303 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:51:19.603094 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
W20251026 08:51:19.604707 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:51:19.604782 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:51:19.612365 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:19.612622 413 tablet_replica.cc:333] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.612833 413 raft_consensus.cc:2243] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.613101 413 raft_consensus.cc:2272] T ee699deac8b34f91849c02a7ff887bce P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.613607 413 tablet_replica.cc:333] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.613756 413 raft_consensus.cc:2243] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.613935 413 raft_consensus.cc:2272] T f94cd0189fb94eefb38bfb4a682ed642 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.614384 413 tablet_replica.cc:333] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.614532 413 raft_consensus.cc:2243] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.614655 413 raft_consensus.cc:2272] T 3f2c155adb5c41bc896579a045c82065 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.615072 413 tablet_replica.cc:333] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.615208 413 raft_consensus.cc:2243] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.615481 413 raft_consensus.cc:2272] T 8eea5573f15d4564aaef98006620de06 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.615850 413 tablet_replica.cc:333] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.616003 413 raft_consensus.cc:2243] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.616178 413 raft_consensus.cc:2272] T 891cf73ca2ac4592aa22fa81a6a817f2 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.616803 413 tablet_replica.cc:333] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.616923 413 raft_consensus.cc:2243] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.617100 413 raft_consensus.cc:2272] T 0d0536d698924dcfb5d376bc8089b17a P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.617826 413 tablet_replica.cc:333] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.618002 413 raft_consensus.cc:2243] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.618338 413 raft_consensus.cc:2272] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.618954 413 tablet_replica.cc:333] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec: stopping tablet replica
I20251026 08:51:19.619074 413 raft_consensus.cc:2243] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.619235 413 raft_consensus.cc:2272] T a8a1daf9a2814a4a9ba917d0896ae4dc P fe20ee26d64d4ca6827c9dd40d3c82ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.624390 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:51:19.628744 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
W20251026 08:51:19.632023 1021 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:51:19.632143 1021 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:51:19.637586 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:19.637820 413 tablet_replica.cc:333] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.637984 413 raft_consensus.cc:2243] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.638164 413 raft_consensus.cc:2272] T 8eea5573f15d4564aaef98006620de06 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.638594 413 tablet_replica.cc:333] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.638749 413 raft_consensus.cc:2243] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.638903 413 raft_consensus.cc:2272] T ee699deac8b34f91849c02a7ff887bce P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.639313 413 tablet_replica.cc:333] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.639454 413 raft_consensus.cc:2243] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.639616 413 raft_consensus.cc:2272] T f94cd0189fb94eefb38bfb4a682ed642 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.639973 413 tablet_replica.cc:333] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.640108 413 raft_consensus.cc:2243] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.640307 413 raft_consensus.cc:2272] T 3f2c155adb5c41bc896579a045c82065 P 146b811db9b842a29796d93521f78516 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.640686 413 tablet_replica.cc:333] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.640805 413 raft_consensus.cc:2243] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.641055 413 raft_consensus.cc:2272] T 891cf73ca2ac4592aa22fa81a6a817f2 P 146b811db9b842a29796d93521f78516 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.641615 413 tablet_replica.cc:333] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.641755 413 raft_consensus.cc:2243] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.641923 413 raft_consensus.cc:2272] T 7c8ae636b1dc4d2fbacb1c36993f04e3 P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.642711 413 tablet_replica.cc:333] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.642879 413 raft_consensus.cc:2243] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.643070 413 raft_consensus.cc:2272] T 0d0536d698924dcfb5d376bc8089b17a P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.643632 413 tablet_replica.cc:333] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516: stopping tablet replica
I20251026 08:51:19.643776 413 raft_consensus.cc:2243] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:19.643926 413 raft_consensus.cc:2272] T a8a1daf9a2814a4a9ba917d0896ae4dc P 146b811db9b842a29796d93521f78516 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.658737 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:51:19.662408 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:51:19.670045 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:19.682588 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:51:19.684669 413 master.cc:561] Master@127.0.103.126:36713 shutting down...
I20251026 08:51:19.691578 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:19.691771 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:19.691890 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 304fd43c9b094d91b7353751368098ec: stopping tablet replica
I20251026 08:51:19.695353 413 master.cc:583] Master@127.0.103.126:36713 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (7443 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20251026 08:51:19.710184 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:45019,127.0.103.125:36279,127.0.103.124:43747
I20251026 08:51:19.710672 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:19.712769 1448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.712881 1449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.713063 1451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:19.713342 413 server_base.cc:1047] running on GCE node
I20251026 08:51:19.713557 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:19.713608 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:19.713653 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468679713631 us; error 0 us; skew 500 ppm
I20251026 08:51:19.713758 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:19.714745 413 webserver.cc:492] Webserver started at http://127.0.103.126:45341/ using document root <none> and password file <none>
I20251026 08:51:19.714906 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:19.714972 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:19.715058 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:19.715543 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "e9e39d0eed7245988b9b0490d8470e10"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.717559 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20251026 08:51:19.719024 1456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:19.719292 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:51:19.719369 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "e9e39d0eed7245988b9b0490d8470e10"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.719461 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:19.753964 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:19.754498 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:19.767112 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:45019
I20251026 08:51:19.767150 1507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:45019 every 8 connection(s)
I20251026 08:51:19.768345 1508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:19.768421 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:19.770558 1510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.770907 1511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.771798 1513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:19.771971 1508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.772074 413 server_base.cc:1047] running on GCE node
I20251026 08:51:19.772598 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:19.772712 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:19.772795 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468679772754 us; error 0 us; skew 500 ppm
I20251026 08:51:19.772989 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:19.774721 413 webserver.cc:492] Webserver started at http://127.0.103.125:42019/ using document root <none> and password file <none>
I20251026 08:51:19.774909 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:19.774991 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:19.775126 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:19.775715 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root/instance:
uuid: "c287d03c0e974d16b6a48bc5a02ac9e5"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.778159 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251026 08:51:19.779394 1508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.780063 1520 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:19.780402 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251026 08:51:19.780540 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root
uuid: "c287d03c0e974d16b6a48bc5a02ac9e5"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.780668 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251026 08:51:19.781260 1508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.103.125:36279: Network error: Client connection negotiation failed: client connection to 127.0.103.125:36279: connect: Connection refused (error 111)
I20251026 08:51:19.793401 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:19.793817 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:19.804688 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.125:36279
I20251026 08:51:19.804735 1572 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.125:36279 every 8 connection(s)
I20251026 08:51:19.805804 1573 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:19.805822 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:19.807650 1575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.807922 1576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.808563 1578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:19.809294 1573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.810451 413 server_base.cc:1047] running on GCE node
I20251026 08:51:19.810726 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:19.810796 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:19.810878 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468679810831 us; error 0 us; skew 500 ppm
I20251026 08:51:19.811026 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:19.812044 413 webserver.cc:492] Webserver started at http://127.0.103.124:41561/ using document root <none> and password file <none>
I20251026 08:51:19.812252 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:19.812359 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:19.812469 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:19.812973 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root/instance:
uuid: "0f9eaeb5a50d41688108690243fa23f9"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.814958 1573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.816362 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 08:51:19.818307 1585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:19.818776 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:19.818883 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root
uuid: "0f9eaeb5a50d41688108690243fa23f9"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.818996 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:19.821573 1573 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } has no permanent_uuid. Determining permanent_uuid...
W20251026 08:51:19.823208 1573 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.103.124:43747: Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:19.837021 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:19.837512 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:19.838479 1508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } attempt: 1
I20251026 08:51:19.844655 1508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } has no permanent_uuid. Determining permanent_uuid...
W20251026 08:51:19.846228 1508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.103.124:43747: Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:19.851090 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.124:43747
I20251026 08:51:19.851135 1637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.124:43747 every 8 connection(s)
I20251026 08:51:19.851923 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:51:19.852238 1638 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:19.854353 1638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.859069 1638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.863689 1638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } has no permanent_uuid. Determining permanent_uuid...
I20251026 08:51:19.870220 1638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9: Bootstrap starting.
I20251026 08:51:19.871419 1638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:19.872695 1638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9: No bootstrap required, opened a new log
I20251026 08:51:19.873179 1638 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:19.873303 1638 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:19.873353 1638 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f9eaeb5a50d41688108690243fa23f9, State: Initialized, Role: FOLLOWER
I20251026 08:51:19.873544 1638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:19.873965 1643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:19.874164 1643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:19.874238 1638 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: configured and running, proceeding with master startup.
W20251026 08:51:19.878352 1654 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 08:51:19.878439 1654 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 08:51:19.879379 1573 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } attempt: 1
I20251026 08:51:19.885378 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: Bootstrap starting.
I20251026 08:51:19.886410 1573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:19.887604 1573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: No bootstrap required, opened a new log
I20251026 08:51:19.888132 1573 raft_consensus.cc:359] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:19.888289 1573 raft_consensus.cc:385] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:19.888365 1573 raft_consensus.cc:740] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c287d03c0e974d16b6a48bc5a02ac9e5, State: Initialized, Role: FOLLOWER
I20251026 08:51:19.888583 1573 consensus_queue.cc:260] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:19.889043 1656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:19.889173 1656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:19.889245 1573 sys_catalog.cc:565] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: configured and running, proceeding with master startup.
W20251026 08:51:19.892671 1667 catalog_manager.cc:1568] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 08:51:19.892743 1667 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251026 08:51:19.911435 1508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } attempt: 1
I20251026 08:51:19.918180 1508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: Bootstrap starting.
I20251026 08:51:19.919344 1508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:19.920892 1508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: No bootstrap required, opened a new log
I20251026 08:51:19.921429 1508 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:19.921584 1508 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:19.921658 1508 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e39d0eed7245988b9b0490d8470e10, State: Initialized, Role: FOLLOWER
I20251026 08:51:19.921869 1508 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:19.922351 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:19.922501 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:19.922580 1508 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:19.925952 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20251026 08:51:19.926050 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20251026 08:51:19.926743 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:19.927016 1680 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251026 08:51:19.927101 1680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20251026 08:51:19.929538 1681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.929586 1682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.930217 1684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:19.930912 413 server_base.cc:1047] running on GCE node
I20251026 08:51:19.931159 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:19.931232 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:19.931274 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468679931258 us; error 0 us; skew 500 ppm
I20251026 08:51:19.931420 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:19.932349 413 webserver.cc:492] Webserver started at http://127.0.103.65:36781/ using document root <none> and password file <none>
I20251026 08:51:19.932984 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:19.933053 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:19.933185 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:19.933715 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "a0ac27c343f847208233b71da70eb26e"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.935848 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20251026 08:51:19.937168 1689 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:19.937544 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:19.937630 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "a0ac27c343f847208233b71da70eb26e"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.937722 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:19.947906 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:19.948339 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:19.948812 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:19.949549 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:19.949615 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:19.949674 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:19.950202 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:19.965106 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:34491
I20251026 08:51:19.965139 1751 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:34491 every 8 connection(s)
I20251026 08:51:19.967269 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:19.969911 1758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:19.986295 1759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:19.986371 1754 heartbeater.cc:344] Connected to a master server at 127.0.103.125:36279
I20251026 08:51:19.986583 1754 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:19.987058 1754 heartbeater.cc:507] Master 127.0.103.125:36279 requested a full tablet report, sending...
I20251026 08:51:19.988345 413 server_base.cc:1047] running on GCE node
I20251026 08:51:19.988687 1538 ts_manager.cc:194] Registered new tserver with Master: a0ac27c343f847208233b71da70eb26e (127.0.103.65:34491)
W20251026 08:51:19.989950 1761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:19.990101 1752 heartbeater.cc:344] Connected to a master server at 127.0.103.124:43747
I20251026 08:51:19.990242 1752 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:19.990453 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:19.990543 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:19.990628 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468679990576 us; error 0 us; skew 500 ppm
I20251026 08:51:19.990650 1752 heartbeater.cc:507] Master 127.0.103.124:43747 requested a full tablet report, sending...
I20251026 08:51:19.990799 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:19.991447 1753 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45019
I20251026 08:51:19.991580 1753 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:19.991797 1603 ts_manager.cc:194] Registered new tserver with Master: a0ac27c343f847208233b71da70eb26e (127.0.103.65:34491)
I20251026 08:51:19.991954 1753 heartbeater.cc:507] Master 127.0.103.126:45019 requested a full tablet report, sending...
I20251026 08:51:19.992868 1473 ts_manager.cc:194] Registered new tserver with Master: a0ac27c343f847208233b71da70eb26e (127.0.103.65:34491)
I20251026 08:51:19.995081 413 webserver.cc:492] Webserver started at http://127.0.103.66:43049/ using document root <none> and password file <none>
I20251026 08:51:19.995255 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:19.995322 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:19.995409 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:19.995958 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "dafb46f69a1844ecaa8e5c3ada6203b5"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:19.998188 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:51:19.999758 1767 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:20.000061 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:20.000150 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "dafb46f69a1844ecaa8e5c3ada6203b5"
format_stamp: "Formatted at 2025-10-26 08:51:19 on dist-test-slave-rd83"
I20251026 08:51:20.000262 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:20.011823 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:20.012290 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:20.012723 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:20.013834 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:20.013890 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:20.013953 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:20.013991 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:20.029583 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:39677
I20251026 08:51:20.029620 1829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:39677 every 8 connection(s)
I20251026 08:51:20.032187 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 08:51:20.040271 1830 heartbeater.cc:344] Connected to a master server at 127.0.103.124:43747
I20251026 08:51:20.040423 1830 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:20.040884 1831 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45019
I20251026 08:51:20.041034 1831 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:20.041347 1830 heartbeater.cc:507] Master 127.0.103.124:43747 requested a full tablet report, sending...
I20251026 08:51:20.041886 1831 heartbeater.cc:507] Master 127.0.103.126:45019 requested a full tablet report, sending...
I20251026 08:51:20.042430 1603 ts_manager.cc:194] Registered new tserver with Master: dafb46f69a1844ecaa8e5c3ada6203b5 (127.0.103.66:39677)
I20251026 08:51:20.042778 1473 ts_manager.cc:194] Registered new tserver with Master: dafb46f69a1844ecaa8e5c3ada6203b5 (127.0.103.66:39677)
I20251026 08:51:20.043645 1832 heartbeater.cc:344] Connected to a master server at 127.0.103.125:36279
I20251026 08:51:20.043848 1832 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:20.044494 1832 heartbeater.cc:507] Master 127.0.103.125:36279 requested a full tablet report, sending...
W20251026 08:51:20.044837 1838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:20.044994 1837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:20.045620 1538 ts_manager.cc:194] Registered new tserver with Master: dafb46f69a1844ecaa8e5c3ada6203b5 (127.0.103.66:39677)
I20251026 08:51:20.046207 413 server_base.cc:1047] running on GCE node
W20251026 08:51:20.046427 1840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:20.046734 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:20.046823 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:20.046886 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468680046865 us; error 0 us; skew 500 ppm
I20251026 08:51:20.047036 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:20.048147 413 webserver.cc:492] Webserver started at http://127.0.103.67:35985/ using document root <none> and password file <none>
I20251026 08:51:20.048398 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:20.048480 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:20.048645 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:20.049247 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "c7a0372891994623a28375d54e0efb2c"
format_stamp: "Formatted at 2025-10-26 08:51:20 on dist-test-slave-rd83"
I20251026 08:51:20.051600 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:51:20.053105 1845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:20.053927 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:20.054035 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "c7a0372891994623a28375d54e0efb2c"
format_stamp: "Formatted at 2025-10-26 08:51:20 on dist-test-slave-rd83"
I20251026 08:51:20.054194 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:20.059682 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:20.060129 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:20.060642 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:20.061404 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:20.061483 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:20.061568 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:20.061646 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:20.079550 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:43867
I20251026 08:51:20.079602 1907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:43867 every 8 connection(s)
I20251026 08:51:20.089201 1908 heartbeater.cc:344] Connected to a master server at 127.0.103.124:43747
I20251026 08:51:20.089342 1908 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:20.089757 1909 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45019
I20251026 08:51:20.089823 1908 heartbeater.cc:507] Master 127.0.103.124:43747 requested a full tablet report, sending...
I20251026 08:51:20.089923 1909 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:20.090395 1909 heartbeater.cc:507] Master 127.0.103.126:45019 requested a full tablet report, sending...
I20251026 08:51:20.090951 1603 ts_manager.cc:194] Registered new tserver with Master: c7a0372891994623a28375d54e0efb2c (127.0.103.67:43867)
I20251026 08:51:20.091204 1910 heartbeater.cc:344] Connected to a master server at 127.0.103.125:36279
I20251026 08:51:20.091837 1910 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:20.091851 1473 ts_manager.cc:194] Registered new tserver with Master: c7a0372891994623a28375d54e0efb2c (127.0.103.67:43867)
I20251026 08:51:20.092180 1910 heartbeater.cc:507] Master 127.0.103.125:36279 requested a full tablet report, sending...
I20251026 08:51:20.092885 1538 ts_manager.cc:194] Registered new tserver with Master: c7a0372891994623a28375d54e0efb2c (127.0.103.67:43867)
I20251026 08:51:20.093647 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.012588986s
I20251026 08:51:20.129254 1643 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:20.129411 1643 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:20.129972 1643 leader_election.cc:290] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019), c287d03c0e974d16b6a48bc5a02ac9e5 (127.0.103.125:36279)
I20251026 08:51:20.130446 1548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f9eaeb5a50d41688108690243fa23f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" is_pre_election: true
I20251026 08:51:20.130422 1483 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f9eaeb5a50d41688108690243fa23f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e39d0eed7245988b9b0490d8470e10" is_pre_election: true
I20251026 08:51:20.130702 1548 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9eaeb5a50d41688108690243fa23f9 in term 0.
I20251026 08:51:20.130703 1483 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f9eaeb5a50d41688108690243fa23f9 in term 0.
I20251026 08:51:20.131080 1586 leader_election.cc:304] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9eaeb5a50d41688108690243fa23f9, c287d03c0e974d16b6a48bc5a02ac9e5; no voters:
I20251026 08:51:20.131286 1643 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:20.131371 1643 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:20.131449 1643 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:20.133040 1643 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:20.133452 1643 leader_election.cc:290] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [CANDIDATE]: Term 1 election: Requested vote from peers e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019), c287d03c0e974d16b6a48bc5a02ac9e5 (127.0.103.125:36279)
I20251026 08:51:20.133741 1483 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f9eaeb5a50d41688108690243fa23f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e39d0eed7245988b9b0490d8470e10"
I20251026 08:51:20.133860 1483 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:20.133901 1548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f9eaeb5a50d41688108690243fa23f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5"
I20251026 08:51:20.134088 1548 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:20.135641 1483 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9eaeb5a50d41688108690243fa23f9 in term 1.
I20251026 08:51:20.135780 1548 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f9eaeb5a50d41688108690243fa23f9 in term 1.
I20251026 08:51:20.135950 1586 leader_election.cc:304] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f9eaeb5a50d41688108690243fa23f9, e9e39d0eed7245988b9b0490d8470e10; no voters:
I20251026 08:51:20.136174 1643 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:20.136498 1643 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 1 LEADER]: Becoming Leader. State: Replica: 0f9eaeb5a50d41688108690243fa23f9, State: Running, Role: LEADER
I20251026 08:51:20.136718 1643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:20.137820 1916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9eaeb5a50d41688108690243fa23f9. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.138128 1916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:20.138767 1918 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:51:20.140373 1918 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:51:20.142699 1483 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Refusing update from remote peer 0f9eaeb5a50d41688108690243fa23f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:20.142695 1548 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Refusing update from remote peer 0f9eaeb5a50d41688108690243fa23f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:20.143128 1916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:20.143326 1643 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:20.146561 1656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9eaeb5a50d41688108690243fa23f9. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.146733 1656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:20.148344 1916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.148598 1916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:20.149271 1656 sys_catalog.cc:455] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.149428 1656 sys_catalog.cc:458] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:20.149782 1918 catalog_manager.cc:1357] Generated new cluster ID: e86ec406fa6648a6a2bcd03817bb3b0e
I20251026 08:51:20.149886 1918 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:51:20.155764 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f9eaeb5a50d41688108690243fa23f9. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.155987 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:20.156844 1916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.157023 1916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:20.158085 1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f9eaeb5a50d41688108690243fa23f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:20.158231 1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [sys.catalog]: This master's current role is: FOLLOWER
I20251026 08:51:20.163844 1918 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:51:20.164696 1918 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:51:20.172299 1918 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9: Generated new TSK 0
I20251026 08:51:20.172483 1918 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:51:20.894011 1667 catalog_manager.cc:1269] Loaded cluster ID: e86ec406fa6648a6a2bcd03817bb3b0e
I20251026 08:51:20.894135 1667 catalog_manager.cc:1562] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: loading cluster ID for follower catalog manager: success
I20251026 08:51:20.896016 1667 catalog_manager.cc:1584] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: acquiring CA information for follower catalog manager: success
I20251026 08:51:20.896893 1667 catalog_manager.cc:1612] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 08:51:20.928165 1680 catalog_manager.cc:1269] Loaded cluster ID: e86ec406fa6648a6a2bcd03817bb3b0e
I20251026 08:51:20.928264 1680 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: loading cluster ID for follower catalog manager: success
I20251026 08:51:20.929898 1680 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: acquiring CA information for follower catalog manager: success
I20251026 08:51:20.930807 1680 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251026 08:51:20.994264 1603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37130
I20251026 08:51:21.044986 1603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37132
I20251026 08:51:21.093137 1603 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37146
I20251026 08:51:21.877681 1653 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:21.995361 1752 heartbeater.cc:499] Master 127.0.103.124:43747 was elected leader, sending a full tablet report...
I20251026 08:51:22.045924 1830 heartbeater.cc:499] Master 127.0.103.124:43747 was elected leader, sending a full tablet report...
I20251026 08:51:22.094264 1908 heartbeater.cc:499] Master 127.0.103.124:43747 was elected leader, sending a full tablet report...
I20251026 08:51:22.251922 413 test_util.cc:276] Using random seed: -485052800
I20251026 08:51:22.266376 1603 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37162:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 08:51:22.294977 1795 tablet_service.cc:1505] Processing CreateTablet for tablet 819176575fb246be8ded27d189d40f74 (DEFAULT_TABLE table=test-workload [id=74731e55622c4684b1ab3cf4ffad9481]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 08:51:22.295585 1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 819176575fb246be8ded27d189d40f74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:22.295487 1717 tablet_service.cc:1505] Processing CreateTablet for tablet 4b93f20db9ab405394c2d8dafbc7d89e (DEFAULT_TABLE table=test-workload [id=74731e55622c4684b1ab3cf4ffad9481]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 08:51:22.295897 1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b93f20db9ab405394c2d8dafbc7d89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:22.300937 1957 tablet_bootstrap.cc:492] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5: Bootstrap starting.
I20251026 08:51:22.301090 1958 tablet_bootstrap.cc:492] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e: Bootstrap starting.
I20251026 08:51:22.301239 1873 tablet_service.cc:1505] Processing CreateTablet for tablet eda345ebffac458d86a7e18b6bf692c4 (DEFAULT_TABLE table=test-workload [id=74731e55622c4684b1ab3cf4ffad9481]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 08:51:22.301712 1873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eda345ebffac458d86a7e18b6bf692c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:22.302776 1957 tablet_bootstrap.cc:654] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:22.303126 1958 tablet_bootstrap.cc:654] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:22.304395 1957 tablet_bootstrap.cc:492] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5: No bootstrap required, opened a new log
I20251026 08:51:22.304500 1957 ts_tablet_manager.cc:1403] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251026 08:51:22.304986 1958 tablet_bootstrap.cc:492] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e: No bootstrap required, opened a new log
I20251026 08:51:22.305125 1958 ts_tablet_manager.cc:1403] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251026 08:51:22.305073 1957 raft_consensus.cc:359] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb46f69a1844ecaa8e5c3ada6203b5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39677 } }
I20251026 08:51:22.305253 1957 raft_consensus.cc:385] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:22.305359 1957 raft_consensus.cc:740] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dafb46f69a1844ecaa8e5c3ada6203b5, State: Initialized, Role: FOLLOWER
I20251026 08:51:22.305567 1957 consensus_queue.cc:260] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb46f69a1844ecaa8e5c3ada6203b5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39677 } }
I20251026 08:51:22.305720 1957 raft_consensus.cc:399] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:22.305776 1957 raft_consensus.cc:493] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:22.305846 1957 raft_consensus.cc:3060] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:22.305941 1958 raft_consensus.cc:359] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ac27c343f847208233b71da70eb26e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34491 } }
I20251026 08:51:22.306565 1958 raft_consensus.cc:385] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:22.306663 1958 raft_consensus.cc:740] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ac27c343f847208233b71da70eb26e, State: Initialized, Role: FOLLOWER
I20251026 08:51:22.306864 1958 consensus_queue.cc:260] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ac27c343f847208233b71da70eb26e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34491 } }
I20251026 08:51:22.307003 1958 raft_consensus.cc:399] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:22.307071 1958 raft_consensus.cc:493] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:22.307158 1958 raft_consensus.cc:3060] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:22.308012 1957 raft_consensus.cc:515] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb46f69a1844ecaa8e5c3ada6203b5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39677 } }
I20251026 08:51:22.308207 1961 tablet_bootstrap.cc:492] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c: Bootstrap starting.
I20251026 08:51:22.308327 1957 leader_election.cc:304] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dafb46f69a1844ecaa8e5c3ada6203b5; no voters:
I20251026 08:51:22.308820 1957 leader_election.cc:290] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:22.308981 1962 raft_consensus.cc:2804] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:22.309914 1958 raft_consensus.cc:515] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ac27c343f847208233b71da70eb26e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34491 } }
I20251026 08:51:22.310114 1958 leader_election.cc:304] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ac27c343f847208233b71da70eb26e; no voters:
I20251026 08:51:22.311416 1961 tablet_bootstrap.cc:654] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:22.313619 1958 leader_election.cc:290] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:22.313792 1965 raft_consensus.cc:2804] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:22.315896 1957 ts_tablet_manager.cc:1434] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5: Time spent starting tablet: real 0.011s user 0.004s sys 0.006s
I20251026 08:51:22.317696 1958 ts_tablet_manager.cc:1434] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e: Time spent starting tablet: real 0.012s user 0.003s sys 0.008s
I20251026 08:51:22.317941 1962 raft_consensus.cc:697] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 1 LEADER]: Becoming Leader. State: Replica: dafb46f69a1844ecaa8e5c3ada6203b5, State: Running, Role: LEADER
I20251026 08:51:22.318127 1965 raft_consensus.cc:697] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 1 LEADER]: Becoming Leader. State: Replica: a0ac27c343f847208233b71da70eb26e, State: Running, Role: LEADER
I20251026 08:51:22.318315 1962 consensus_queue.cc:237] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb46f69a1844ecaa8e5c3ada6203b5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39677 } }
I20251026 08:51:22.318585 1965 consensus_queue.cc:237] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ac27c343f847208233b71da70eb26e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34491 } }
I20251026 08:51:22.321198 1961 tablet_bootstrap.cc:492] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c: No bootstrap required, opened a new log
I20251026 08:51:22.321447 1961 ts_tablet_manager.cc:1403] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20251026 08:51:22.321498 1603 catalog_manager.cc:5649] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e reported cstate change: term changed from 0 to 1, leader changed from <none> to a0ac27c343f847208233b71da70eb26e (127.0.103.65). New cstate: current_term: 1 leader_uuid: "a0ac27c343f847208233b71da70eb26e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ac27c343f847208233b71da70eb26e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 34491 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:22.322572 1961 raft_consensus.cc:359] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a0372891994623a28375d54e0efb2c" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43867 } }
I20251026 08:51:22.322777 1961 raft_consensus.cc:385] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:22.322892 1961 raft_consensus.cc:740] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7a0372891994623a28375d54e0efb2c, State: Initialized, Role: FOLLOWER
I20251026 08:51:22.323168 1961 consensus_queue.cc:260] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a0372891994623a28375d54e0efb2c" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43867 } }
I20251026 08:51:22.323374 1961 raft_consensus.cc:399] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:22.323484 1961 raft_consensus.cc:493] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:22.323625 1961 raft_consensus.cc:3060] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:22.326762 1961 raft_consensus.cc:515] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a0372891994623a28375d54e0efb2c" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43867 } }
I20251026 08:51:22.326996 1961 leader_election.cc:304] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7a0372891994623a28375d54e0efb2c; no voters:
I20251026 08:51:22.329566 1602 catalog_manager.cc:5649] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to dafb46f69a1844ecaa8e5c3ada6203b5 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "dafb46f69a1844ecaa8e5c3ada6203b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dafb46f69a1844ecaa8e5c3ada6203b5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 39677 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:22.331852 1961 leader_election.cc:290] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:22.332006 1972 raft_consensus.cc:2804] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:22.336747 1972 raft_consensus.cc:697] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 1 LEADER]: Becoming Leader. State: Replica: c7a0372891994623a28375d54e0efb2c, State: Running, Role: LEADER
I20251026 08:51:22.337072 1972 consensus_queue.cc:237] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a0372891994623a28375d54e0efb2c" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43867 } }
I20251026 08:51:22.340499 1961 ts_tablet_manager.cc:1434] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c: Time spent starting tablet: real 0.019s user 0.007s sys 0.006s
I20251026 08:51:22.344195 1602 catalog_manager.cc:5649] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c reported cstate change: term changed from 0 to 1, leader changed from <none> to c7a0372891994623a28375d54e0efb2c (127.0.103.67). New cstate: current_term: 1 leader_uuid: "c7a0372891994623a28375d54e0efb2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7a0372891994623a28375d54e0efb2c" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 43867 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:22.358448 413 master.cc:561] Master@127.0.103.124:43747 shutting down...
I20251026 08:51:22.364333 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:22.364598 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:22.364702 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 0f9eaeb5a50d41688108690243fa23f9: stopping tablet replica
I20251026 08:51:22.369371 413 master.cc:583] Master@127.0.103.124:43747 shutdown complete.
W20251026 08:51:23.343850 1752 heartbeater.cc:646] Failed to heartbeat to 127.0.103.124:43747 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:23.853928 1981 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f9eaeb5a50d41688108690243fa23f9)
I20251026 08:51:23.854142 1981 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:23.854310 1982 raft_consensus.cc:493] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0f9eaeb5a50d41688108690243fa23f9)
I20251026 08:51:23.854436 1982 raft_consensus.cc:515] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:23.854921 1981 leader_election.cc:290] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c287d03c0e974d16b6a48bc5a02ac9e5 (127.0.103.125:36279), 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747)
I20251026 08:51:23.855067 1982 leader_election.cc:290] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019), 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747)
I20251026 08:51:23.855715 1483 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "e9e39d0eed7245988b9b0490d8470e10" is_pre_election: true
I20251026 08:51:23.855909 1483 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c287d03c0e974d16b6a48bc5a02ac9e5 in term 1.
I20251026 08:51:23.856118 1548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9e39d0eed7245988b9b0490d8470e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" is_pre_election: true
I20251026 08:51:23.856308 1522 leader_election.cc:304] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c287d03c0e974d16b6a48bc5a02ac9e5, e9e39d0eed7245988b9b0490d8470e10; no voters:
I20251026 08:51:23.856336 1548 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9e39d0eed7245988b9b0490d8470e10 in term 1.
I20251026 08:51:23.856586 1982 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:51:23.856870 1457 leader_election.cc:304] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c287d03c0e974d16b6a48bc5a02ac9e5, e9e39d0eed7245988b9b0490d8470e10; no voters:
I20251026 08:51:23.857151 1981 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:51:23.856745 1982 raft_consensus.cc:493] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f9eaeb5a50d41688108690243fa23f9)
I20251026 08:51:23.857342 1981 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0f9eaeb5a50d41688108690243fa23f9)
I20251026 08:51:23.857347 1982 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:23.857467 1981 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 1 FOLLOWER]: Advancing to term 2
W20251026 08:51:23.857700 1457 leader_election.cc:336] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747): Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
W20251026 08:51:23.858127 1522 leader_election.cc:336] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747): Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:23.859656 1982 raft_consensus.cc:515] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:23.859696 1981 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:23.860143 1982 leader_election.cc:290] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 2 election: Requested vote from peers e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019), 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747)
I20251026 08:51:23.860252 1981 leader_election.cc:290] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [CANDIDATE]: Term 2 election: Requested vote from peers c287d03c0e974d16b6a48bc5a02ac9e5 (127.0.103.125:36279), 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747)
I20251026 08:51:23.860491 1483 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "e9e39d0eed7245988b9b0490d8470e10"
I20251026 08:51:23.860709 1483 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c287d03c0e974d16b6a48bc5a02ac9e5 in current term 2: Already voted for candidate e9e39d0eed7245988b9b0490d8470e10 in this term.
I20251026 08:51:23.860827 1548 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9e39d0eed7245988b9b0490d8470e10" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5"
I20251026 08:51:23.861011 1548 raft_consensus.cc:2393] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e9e39d0eed7245988b9b0490d8470e10 in current term 2: Already voted for candidate c287d03c0e974d16b6a48bc5a02ac9e5 in this term.
W20251026 08:51:23.861774 1522 leader_election.cc:336] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747): Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
W20251026 08:51:23.861795 1457 leader_election.cc:336] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747): Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:23.861876 1522 leader_election.cc:304] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c287d03c0e974d16b6a48bc5a02ac9e5; no voters: 0f9eaeb5a50d41688108690243fa23f9, e9e39d0eed7245988b9b0490d8470e10
I20251026 08:51:23.861932 1457 leader_election.cc:304] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9e39d0eed7245988b9b0490d8470e10; no voters: 0f9eaeb5a50d41688108690243fa23f9, c287d03c0e974d16b6a48bc5a02ac9e5
I20251026 08:51:23.862135 1982 raft_consensus.cc:2749] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 08:51:23.862231 1981 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 08:51:25.611968 1993 raft_consensus.cc:493] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:25.612154 1993 raft_consensus.cc:515] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:25.612825 1993 leader_election.cc:290] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019), 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747)
I20251026 08:51:25.613442 1483 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "e9e39d0eed7245988b9b0490d8470e10" is_pre_election: true
I20251026 08:51:25.613646 1483 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c287d03c0e974d16b6a48bc5a02ac9e5 in term 2.
W20251026 08:51:25.614038 1522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20251026 08:51:25.618433 1522 leader_election.cc:304] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c287d03c0e974d16b6a48bc5a02ac9e5, e9e39d0eed7245988b9b0490d8470e10; no voters:
I20251026 08:51:25.618695 1993 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 08:51:25.618821 1993 raft_consensus.cc:493] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:25.618911 1993 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 2 FOLLOWER]: Advancing to term 3
W20251026 08:51:25.619233 1522 leader_election.cc:336] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747): Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:25.620848 1993 raft_consensus.cc:515] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:25.621309 1993 leader_election.cc:290] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 3 election: Requested vote from peers e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019), 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747)
I20251026 08:51:25.621783 1483 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" candidate_term: 3 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "e9e39d0eed7245988b9b0490d8470e10"
I20251026 08:51:25.621963 1483 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 2 FOLLOWER]: Advancing to term 3
W20251026 08:51:25.622658 1522 leader_election.cc:336] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0f9eaeb5a50d41688108690243fa23f9 (127.0.103.124:43747): Network error: Client connection negotiation failed: client connection to 127.0.103.124:43747: connect: Connection refused (error 111)
I20251026 08:51:25.623870 1483 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c287d03c0e974d16b6a48bc5a02ac9e5 in term 3.
I20251026 08:51:25.624225 1522 leader_election.cc:304] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c287d03c0e974d16b6a48bc5a02ac9e5, e9e39d0eed7245988b9b0490d8470e10; no voters: 0f9eaeb5a50d41688108690243fa23f9
I20251026 08:51:25.624507 1993 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:51:25.624857 1993 raft_consensus.cc:697] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 3 LEADER]: Becoming Leader. State: Replica: c287d03c0e974d16b6a48bc5a02ac9e5, State: Running, Role: LEADER
I20251026 08:51:25.625147 1993 consensus_queue.cc:237] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } }
I20251026 08:51:25.626384 1996 sys_catalog.cc:455] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c287d03c0e974d16b6a48bc5a02ac9e5. Latest consensus state: current_term: 3 leader_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9e39d0eed7245988b9b0490d8470e10" member_type: VOTER last_known_addr { host: "127.0.103.126" port: 45019 } } peers { permanent_uuid: "c287d03c0e974d16b6a48bc5a02ac9e5" member_type: VOTER last_known_addr { host: "127.0.103.125" port: 36279 } } peers { permanent_uuid: "0f9eaeb5a50d41688108690243fa23f9" member_type: VOTER last_known_addr { host: "127.0.103.124" port: 43747 } } }
I20251026 08:51:25.626605 1996 sys_catalog.cc:458] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:25.627072 1998 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:51:25.628687 1998 catalog_manager.cc:679] Loaded metadata for table test-workload [id=74731e55622c4684b1ab3cf4ffad9481]
I20251026 08:51:25.629978 1998 tablet_loader.cc:96] loaded metadata for tablet 4b93f20db9ab405394c2d8dafbc7d89e (table test-workload [id=74731e55622c4684b1ab3cf4ffad9481])
I20251026 08:51:25.630342 1998 tablet_loader.cc:96] loaded metadata for tablet 819176575fb246be8ded27d189d40f74 (table test-workload [id=74731e55622c4684b1ab3cf4ffad9481])
I20251026 08:51:25.630656 1998 tablet_loader.cc:96] loaded metadata for tablet eda345ebffac458d86a7e18b6bf692c4 (table test-workload [id=74731e55622c4684b1ab3cf4ffad9481])
I20251026 08:51:25.630889 1998 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:51:25.631680 1998 catalog_manager.cc:1269] Loaded cluster ID: e86ec406fa6648a6a2bcd03817bb3b0e
I20251026 08:51:25.631754 1998 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:51:25.632798 1998 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:51:25.633576 1998 catalog_manager.cc:6033] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: Loaded TSK: 0
I20251026 08:51:25.634198 1998 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:51:25.892632 1666 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:25.892758 1666 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:25.944073 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:51:25.954291 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:25.954605 413 tablet_replica.cc:333] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e: stopping tablet replica
I20251026 08:51:25.954780 413 raft_consensus.cc:2243] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:25.954998 413 raft_consensus.cc:2272] T 4b93f20db9ab405394c2d8dafbc7d89e P a0ac27c343f847208233b71da70eb26e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:25.968595 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:51:25.971311 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:51:25.979138 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:25.979404 413 tablet_replica.cc:333] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5: stopping tablet replica
I20251026 08:51:25.979605 413 raft_consensus.cc:2243] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:25.979759 413 raft_consensus.cc:2272] T 819176575fb246be8ded27d189d40f74 P dafb46f69a1844ecaa8e5c3ada6203b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:25.992815 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:51:25.995441 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:51:26.003532 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:26.003773 413 tablet_replica.cc:333] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c: stopping tablet replica
I20251026 08:51:26.003944 413 raft_consensus.cc:2243] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:26.004106 413 raft_consensus.cc:2272] T eda345ebffac458d86a7e18b6bf692c4 P c7a0372891994623a28375d54e0efb2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:26.017258 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:51:26.020069 413 master.cc:561] Master@127.0.103.126:45019 shutting down...
W20251026 08:51:26.027606 1522 consensus_peers.cc:597] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 -> Peer e9e39d0eed7245988b9b0490d8470e10 (127.0.103.126:45019): Couldn't send request to peer e9e39d0eed7245988b9b0490d8470e10. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:51:26.027894 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:26.028071 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:26.028209 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P e9e39d0eed7245988b9b0490d8470e10: stopping tablet replica
I20251026 08:51:26.041813 413 master.cc:583] Master@127.0.103.126:45019 shutdown complete.
I20251026 08:51:26.045085 413 master.cc:561] Master@127.0.103.125:36279 shutting down...
I20251026 08:51:26.051457 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:51:26.051709 413 pending_rounds.cc:70] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: Trying to abort 1 pending ops.
I20251026 08:51:26.051860 413 pending_rounds.cc:77] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: Aborting op as it isn't in flight: id { term: 3 index: 11 } timestamp: 7214975736322605056 op_type: NO_OP noop_request { }
I20251026 08:51:26.051987 413 raft_consensus.cc:2889] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:51:26.052089 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:26.052242 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P c287d03c0e974d16b6a48bc5a02ac9e5: stopping tablet replica
I20251026 08:51:26.065526 413 master.cc:583] Master@127.0.103.125:36279 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (6366 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20251026 08:51:26.077118 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:45685
I20251026 08:51:26.077593 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:26.079797 2000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.079823 1999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.080200 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:26.080667 413 server_base.cc:1047] running on GCE node
I20251026 08:51:26.080878 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:26.080950 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:26.081002 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468686080986 us; error 0 us; skew 500 ppm
I20251026 08:51:26.081152 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:26.085832 413 webserver.cc:492] Webserver started at http://127.0.103.126:41443/ using document root <none> and password file <none>
I20251026 08:51:26.086055 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:26.086177 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:26.086310 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:26.086848 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "69860859464d44de8576fe622aec1b9e"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.088992 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:51:26.090476 2007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.090807 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:51:26.090898 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "69860859464d44de8576fe622aec1b9e"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.090983 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:26.102514 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:26.103076 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:26.123239 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:45685
I20251026 08:51:26.123274 2058 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:45685 every 8 connection(s)
I20251026 08:51:26.124686 2059 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:26.129626 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e: Bootstrap starting.
I20251026 08:51:26.131158 2059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:26.132834 2059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e: No bootstrap required, opened a new log
I20251026 08:51:26.133513 2059 raft_consensus.cc:359] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69860859464d44de8576fe622aec1b9e" member_type: VOTER }
I20251026 08:51:26.133641 2059 raft_consensus.cc:385] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:26.133721 2059 raft_consensus.cc:740] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69860859464d44de8576fe622aec1b9e, State: Initialized, Role: FOLLOWER
I20251026 08:51:26.133929 2059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69860859464d44de8576fe622aec1b9e" member_type: VOTER }
I20251026 08:51:26.134049 2059 raft_consensus.cc:399] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:26.134148 2059 raft_consensus.cc:493] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:26.134248 2059 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:26.135666 2059 raft_consensus.cc:515] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69860859464d44de8576fe622aec1b9e" member_type: VOTER }
I20251026 08:51:26.135886 2059 leader_election.cc:304] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69860859464d44de8576fe622aec1b9e; no voters:
I20251026 08:51:26.136236 2059 leader_election.cc:290] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:26.136430 2062 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:26.137049 2059 sys_catalog.cc:565] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:26.137027 2062 raft_consensus.cc:697] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 1 LEADER]: Becoming Leader. State: Replica: 69860859464d44de8576fe622aec1b9e, State: Running, Role: LEADER
I20251026 08:51:26.137404 2062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69860859464d44de8576fe622aec1b9e" member_type: VOTER }
I20251026 08:51:26.138514 2063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69860859464d44de8576fe622aec1b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69860859464d44de8576fe622aec1b9e" member_type: VOTER } }
I20251026 08:51:26.138643 2064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69860859464d44de8576fe622aec1b9e. Latest consensus state: current_term: 1 leader_uuid: "69860859464d44de8576fe622aec1b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69860859464d44de8576fe622aec1b9e" member_type: VOTER } }
I20251026 08:51:26.138773 2063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:26.138849 2064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:26.139490 2067 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:51:26.141804 2067 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:51:26.143651 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:51:26.145022 2067 catalog_manager.cc:1357] Generated new cluster ID: 89f437d678bd4c00ad080cbfe774e08a
I20251026 08:51:26.145103 2067 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:51:26.170665 2067 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:51:26.171278 2067 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:51:26.177683 2067 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e: Generated new TSK 0
I20251026 08:51:26.177866 2067 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:51:26.208496 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:26.210956 2080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.211642 2083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.211642 2081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:26.212221 413 server_base.cc:1047] running on GCE node
I20251026 08:51:26.212415 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:26.212474 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:26.212520 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468686212498 us; error 0 us; skew 500 ppm
I20251026 08:51:26.212615 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:26.213587 413 webserver.cc:492] Webserver started at http://127.0.103.65:45949/ using document root <none> and password file <none>
I20251026 08:51:26.213763 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:26.213829 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:26.213925 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:26.214505 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "ab87b138f62f4b8ebb644e0f91683bbf"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.216482 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20251026 08:51:26.217865 2088 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.218246 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:26.218340 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "ab87b138f62f4b8ebb644e0f91683bbf"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.218456 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:26.225373 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:26.225786 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:26.226207 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:26.226907 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:26.226961 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.227033 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:26.227070 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.239267 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:44581
I20251026 08:51:26.239288 2150 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:44581 every 8 connection(s)
I20251026 08:51:26.240813 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:26.243454 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.244032 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.247500 2158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:26.247573 413 server_base.cc:1047] running on GCE node
I20251026 08:51:26.247879 2151 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45685
I20251026 08:51:26.247988 2151 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:26.248157 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:26.248257 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:26.248328 2151 heartbeater.cc:507] Master 127.0.103.126:45685 requested a full tablet report, sending...
I20251026 08:51:26.248376 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468686248308 us; error 0 us; skew 500 ppm
I20251026 08:51:26.248610 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:26.249359 2024 ts_manager.cc:194] Registered new tserver with Master: ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65:44581)
I20251026 08:51:26.249732 413 webserver.cc:492] Webserver started at http://127.0.103.66:32911/ using document root <none> and password file <none>
I20251026 08:51:26.249923 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:26.249991 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:26.250144 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:26.250646 2024 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40578
I20251026 08:51:26.250682 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.252704 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:51:26.253911 2163 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.254226 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:26.254336 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.254432 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:26.273258 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:26.273685 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:26.274060 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:26.274792 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:26.274848 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.274893 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:26.274923 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.285761 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:42673
I20251026 08:51:26.285777 2225 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:42673 every 8 connection(s)
I20251026 08:51:26.287580 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:26.289999 2229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:26.290060 2230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:26.291455 413 server_base.cc:1047] running on GCE node
W20251026 08:51:26.291720 2232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:26.292054 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:26.292124 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:26.292155 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468686292152 us; error 0 us; skew 500 ppm
I20251026 08:51:26.292312 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:26.293159 2226 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45685
I20251026 08:51:26.293264 2226 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:26.293471 413 webserver.cc:492] Webserver started at http://127.0.103.67:38079/ using document root <none> and password file <none>
I20251026 08:51:26.293566 2226 heartbeater.cc:507] Master 127.0.103.126:45685 requested a full tablet report, sending...
I20251026 08:51:26.293666 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:26.293720 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:26.293812 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:26.294353 2024 ts_manager.cc:194] Registered new tserver with Master: a7a8b19dfcb343b0b02c7e4a0b9c84d9 (127.0.103.66:42673)
I20251026 08:51:26.294412 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "dc1db092db97463194f99e71711ed461"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.295224 2024 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40590
I20251026 08:51:26.296949 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:51:26.298226 2237 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.298525 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:26.298610 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "dc1db092db97463194f99e71711ed461"
format_stamp: "Formatted at 2025-10-26 08:51:26 on dist-test-slave-rd83"
I20251026 08:51:26.298710 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:26.304718 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:26.305094 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:26.305545 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:26.306228 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:26.306277 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.306350 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:26.306480 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:26.319994 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:36629
I20251026 08:51:26.320015 2299 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:36629 every 8 connection(s)
I20251026 08:51:26.325944 2300 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45685
I20251026 08:51:26.326095 2300 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:26.326401 2300 heartbeater.cc:507] Master 127.0.103.126:45685 requested a full tablet report, sending...
I20251026 08:51:26.327134 2024 ts_manager.cc:194] Registered new tserver with Master: dc1db092db97463194f99e71711ed461 (127.0.103.67:36629)
I20251026 08:51:26.327689 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006875935s
I20251026 08:51:26.327770 2024 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:40602
I20251026 08:51:27.251784 2151 heartbeater.cc:499] Master 127.0.103.126:45685 was elected leader, sending a full tablet report...
I20251026 08:51:27.296523 2226 heartbeater.cc:499] Master 127.0.103.126:45685 was elected leader, sending a full tablet report...
I20251026 08:51:27.329044 2300 heartbeater.cc:499] Master 127.0.103.126:45685 was elected leader, sending a full tablet report...
I20251026 08:51:27.354559 413 test_util.cc:276] Using random seed: -479950175
I20251026 08:51:27.367801 2024 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46836:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:51:27.368495 2024 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:51:27.383975 2265 tablet_service.cc:1505] Processing CreateTablet for tablet f38d614def11479b80211a50c3c0ce25 (DEFAULT_TABLE table=test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251026 08:51:27.383975 2264 tablet_service.cc:1505] Processing CreateTablet for tablet ad2c20814a3946d2a51700e7541ccbce (DEFAULT_TABLE table=test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251026 08:51:27.384364 2265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38d614def11479b80211a50c3c0ce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:27.384586 2264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2c20814a3946d2a51700e7541ccbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:27.385460 2116 tablet_service.cc:1505] Processing CreateTablet for tablet f38d614def11479b80211a50c3c0ce25 (DEFAULT_TABLE table=test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251026 08:51:27.385793 2116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38d614def11479b80211a50c3c0ce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:27.386080 2190 tablet_service.cc:1505] Processing CreateTablet for tablet ad2c20814a3946d2a51700e7541ccbce (DEFAULT_TABLE table=test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251026 08:51:27.386394 2190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2c20814a3946d2a51700e7541ccbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:27.386447 2191 tablet_service.cc:1505] Processing CreateTablet for tablet f38d614def11479b80211a50c3c0ce25 (DEFAULT_TABLE table=test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20251026 08:51:27.386793 2191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38d614def11479b80211a50c3c0ce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:27.391191 2115 tablet_service.cc:1505] Processing CreateTablet for tablet ad2c20814a3946d2a51700e7541ccbce (DEFAULT_TABLE table=test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20251026 08:51:27.391517 2115 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2c20814a3946d2a51700e7541ccbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:27.397541 2322 tablet_bootstrap.cc:492] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Bootstrap starting.
I20251026 08:51:27.398525 2321 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: Bootstrap starting.
I20251026 08:51:27.399012 2320 tablet_bootstrap.cc:492] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: Bootstrap starting.
I20251026 08:51:27.399670 2322 tablet_bootstrap.cc:654] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:27.400122 2321 tablet_bootstrap.cc:654] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:27.400792 2320 tablet_bootstrap.cc:654] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:27.402755 2320 tablet_bootstrap.cc:492] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: No bootstrap required, opened a new log
I20251026 08:51:27.402892 2320 ts_tablet_manager.cc:1403] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:51:27.403571 2321 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: No bootstrap required, opened a new log
I20251026 08:51:27.403676 2321 ts_tablet_manager.cc:1403] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:51:27.403604 2320 raft_consensus.cc:359] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.403757 2320 raft_consensus.cc:385] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:27.403813 2320 raft_consensus.cc:740] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1db092db97463194f99e71711ed461, State: Initialized, Role: FOLLOWER
I20251026 08:51:27.403831 2322 tablet_bootstrap.cc:492] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9: No bootstrap required, opened a new log
I20251026 08:51:27.403919 2322 ts_tablet_manager.cc:1403] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20251026 08:51:27.403966 2320 consensus_queue.cc:260] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.404285 2321 raft_consensus.cc:359] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.404431 2322 raft_consensus.cc:359] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.404569 2322 raft_consensus.cc:385] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:27.404584 2320 ts_tablet_manager.cc:1434] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:51:27.404619 2322 raft_consensus.cc:740] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a8b19dfcb343b0b02c7e4a0b9c84d9, State: Initialized, Role: FOLLOWER
I20251026 08:51:27.404820 2320 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: Bootstrap starting.
I20251026 08:51:27.404767 2322 consensus_queue.cc:260] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.404417 2321 raft_consensus.cc:385] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:27.405028 2321 raft_consensus.cc:740] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab87b138f62f4b8ebb644e0f91683bbf, State: Initialized, Role: FOLLOWER
I20251026 08:51:27.406490 2320 tablet_bootstrap.cc:654] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:27.407699 2320 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: No bootstrap required, opened a new log
I20251026 08:51:27.407797 2320 ts_tablet_manager.cc:1403] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:51:27.405222 2321 consensus_queue.cc:260] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.408321 2320 raft_consensus.cc:359] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.408573 2320 raft_consensus.cc:385] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:27.408645 2320 raft_consensus.cc:740] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc1db092db97463194f99e71711ed461, State: Initialized, Role: FOLLOWER
I20251026 08:51:27.408818 2320 consensus_queue.cc:260] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.409210 2320 ts_tablet_manager.cc:1434] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:51:27.410579 2321 ts_tablet_manager.cc:1434] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251026 08:51:27.410724 2322 ts_tablet_manager.cc:1434] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20251026 08:51:27.412065 2322 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Bootstrap starting.
I20251026 08:51:27.412122 2321 tablet_bootstrap.cc:492] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf: Bootstrap starting.
I20251026 08:51:27.413514 2322 tablet_bootstrap.cc:654] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:27.414135 2321 tablet_bootstrap.cc:654] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:27.416021 2321 tablet_bootstrap.cc:492] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf: No bootstrap required, opened a new log
I20251026 08:51:27.416168 2321 ts_tablet_manager.cc:1403] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20251026 08:51:27.416931 2321 raft_consensus.cc:359] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.417133 2321 raft_consensus.cc:385] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:27.417202 2321 raft_consensus.cc:740] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab87b138f62f4b8ebb644e0f91683bbf, State: Initialized, Role: FOLLOWER
I20251026 08:51:27.417457 2321 consensus_queue.cc:260] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.417992 2321 ts_tablet_manager.cc:1434] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:51:27.420977 2322 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9: No bootstrap required, opened a new log
I20251026 08:51:27.421082 2322 ts_tablet_manager.cc:1403] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20251026 08:51:27.421551 2322 raft_consensus.cc:359] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.421672 2322 raft_consensus.cc:385] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:27.421733 2322 raft_consensus.cc:740] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a8b19dfcb343b0b02c7e4a0b9c84d9, State: Initialized, Role: FOLLOWER
I20251026 08:51:27.421916 2322 consensus_queue.cc:260] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.422259 2322 ts_tablet_manager.cc:1434] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251026 08:51:27.504146 2326 raft_consensus.cc:493] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:27.504343 2326 raft_consensus.cc:515] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.505174 2326 leader_election.cc:290] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65:44581), a7a8b19dfcb343b0b02c7e4a0b9c84d9 (127.0.103.66:42673)
I20251026 08:51:27.511433 2126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" is_pre_election: true
I20251026 08:51:27.511687 2126 raft_consensus.cc:2468] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 0.
I20251026 08:51:27.511857 2201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" is_pre_election: true
I20251026 08:51:27.512048 2201 raft_consensus.cc:2468] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 0.
I20251026 08:51:27.512055 2238 leader_election.cc:304] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab87b138f62f4b8ebb644e0f91683bbf, dc1db092db97463194f99e71711ed461; no voters:
I20251026 08:51:27.512284 2326 raft_consensus.cc:2804] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:27.512363 2326 raft_consensus.cc:493] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:27.512426 2326 raft_consensus.cc:3060] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:27.513983 2326 raft_consensus.cc:515] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.514420 2326 leader_election.cc:290] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 election: Requested vote from peers ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65:44581), a7a8b19dfcb343b0b02c7e4a0b9c84d9 (127.0.103.66:42673)
I20251026 08:51:27.514729 2126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab87b138f62f4b8ebb644e0f91683bbf"
I20251026 08:51:27.514778 2201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9"
I20251026 08:51:27.514839 2126 raft_consensus.cc:3060] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:27.514904 2201 raft_consensus.cc:3060] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:27.516405 2126 raft_consensus.cc:2468] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 1.
I20251026 08:51:27.516503 2201 raft_consensus.cc:2468] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 1.
I20251026 08:51:27.516701 2238 leader_election.cc:304] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab87b138f62f4b8ebb644e0f91683bbf, dc1db092db97463194f99e71711ed461; no voters:
I20251026 08:51:27.516860 2326 raft_consensus.cc:2804] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:27.517145 2326 raft_consensus.cc:697] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Becoming Leader. State: Replica: dc1db092db97463194f99e71711ed461, State: Running, Role: LEADER
I20251026 08:51:27.517349 2326 consensus_queue.cc:237] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:27.519542 2024 catalog_manager.cc:5649] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc1db092db97463194f99e71711ed461 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "dc1db092db97463194f99e71711ed461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } health_report { overall_health: HEALTHY } } }
I20251026 08:51:27.565068 2326 raft_consensus.cc:493] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:51:27.565210 2326 raft_consensus.cc:515] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.565788 2326 leader_election.cc:290] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65:44581), a7a8b19dfcb343b0b02c7e4a0b9c84d9 (127.0.103.66:42673)
I20251026 08:51:27.566372 2201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f38d614def11479b80211a50c3c0ce25" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" is_pre_election: true
I20251026 08:51:27.566543 2201 raft_consensus.cc:2468] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 0.
I20251026 08:51:27.566628 2126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f38d614def11479b80211a50c3c0ce25" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" is_pre_election: true
I20251026 08:51:27.566818 2126 raft_consensus.cc:2468] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 0.
I20251026 08:51:27.566922 2241 leader_election.cc:304] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7a8b19dfcb343b0b02c7e4a0b9c84d9, dc1db092db97463194f99e71711ed461; no voters:
I20251026 08:51:27.567159 2326 raft_consensus.cc:2804] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:51:27.567229 2326 raft_consensus.cc:493] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:51:27.567282 2326 raft_consensus.cc:3060] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:27.568869 2326 raft_consensus.cc:515] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.569274 2326 leader_election.cc:290] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 election: Requested vote from peers ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65:44581), a7a8b19dfcb343b0b02c7e4a0b9c84d9 (127.0.103.66:42673)
I20251026 08:51:27.569690 2201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f38d614def11479b80211a50c3c0ce25" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9"
I20251026 08:51:27.569733 2126 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f38d614def11479b80211a50c3c0ce25" candidate_uuid: "dc1db092db97463194f99e71711ed461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab87b138f62f4b8ebb644e0f91683bbf"
I20251026 08:51:27.569839 2201 raft_consensus.cc:3060] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:27.569876 2126 raft_consensus.cc:3060] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:27.571532 2126 raft_consensus.cc:2468] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 1.
I20251026 08:51:27.571540 2201 raft_consensus.cc:2468] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc1db092db97463194f99e71711ed461 in term 1.
I20251026 08:51:27.571834 2238 leader_election.cc:304] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a7a8b19dfcb343b0b02c7e4a0b9c84d9, ab87b138f62f4b8ebb644e0f91683bbf, dc1db092db97463194f99e71711ed461; no voters:
I20251026 08:51:27.572057 2326 raft_consensus.cc:2804] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:27.572177 2326 raft_consensus.cc:697] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Becoming Leader. State: Replica: dc1db092db97463194f99e71711ed461, State: Running, Role: LEADER
I20251026 08:51:27.572404 2326 consensus_queue.cc:237] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } }
I20251026 08:51:27.574635 2024 catalog_manager.cc:5649] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc1db092db97463194f99e71711ed461 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "dc1db092db97463194f99e71711ed461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:27.995167 2326 consensus_queue.cc:1048] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:27.999717 2334 consensus_queue.cc:1048] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:28.110707 2326 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:28.114503 2334 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:51:28.143623 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:28.149718 2275 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce"
dest_uuid: "dc1db092db97463194f99e71711ed461"
mode: GRACEFUL
new_leader_uuid: "ab87b138f62f4b8ebb644e0f91683bbf"
from {username='slave'} at 127.0.0.1:39740
I20251026 08:51:28.149880 2275 raft_consensus.cc:606] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Received request to transfer leadership to TS ab87b138f62f4b8ebb644e0f91683bbf
I20251026 08:51:28.150313 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:51:28.150418 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:28.572194 2334 raft_consensus.cc:993] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: : Instructing follower ab87b138f62f4b8ebb644e0f91683bbf to start an election
I20251026 08:51:28.572376 2334 raft_consensus.cc:1081] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Signalling peer ab87b138f62f4b8ebb644e0f91683bbf to start an election
I20251026 08:51:28.572923 2126 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce"
dest_uuid: "ab87b138f62f4b8ebb644e0f91683bbf"
from {username='slave'} at 127.0.0.1:51602
I20251026 08:51:28.573087 2126 raft_consensus.cc:493] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:51:28.573194 2126 raft_consensus.cc:3060] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:28.574950 2126 raft_consensus.cc:515] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:28.575579 2126 leader_election.cc:290] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [CANDIDATE]: Term 2 election: Requested vote from peers a7a8b19dfcb343b0b02c7e4a0b9c84d9 (127.0.103.66:42673), dc1db092db97463194f99e71711ed461 (127.0.103.67:36629)
I20251026 08:51:28.581708 2275 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce" candidate_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dc1db092db97463194f99e71711ed461"
I20251026 08:51:28.581869 2275 raft_consensus.cc:3055] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:51:28.581986 2275 raft_consensus.cc:740] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dc1db092db97463194f99e71711ed461, State: Running, Role: LEADER
I20251026 08:51:28.582036 2201 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad2c20814a3946d2a51700e7541ccbce" candidate_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9"
I20251026 08:51:28.582237 2201 raft_consensus.cc:3060] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:28.582319 2275 consensus_queue.cc:260] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:28.582679 2275 raft_consensus.cc:3060] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:51:28.584797 2201 raft_consensus.cc:2468] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab87b138f62f4b8ebb644e0f91683bbf in term 2.
I20251026 08:51:28.585032 2275 raft_consensus.cc:2468] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab87b138f62f4b8ebb644e0f91683bbf in term 2.
I20251026 08:51:28.585230 2092 leader_election.cc:304] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7a8b19dfcb343b0b02c7e4a0b9c84d9, ab87b138f62f4b8ebb644e0f91683bbf; no voters:
I20251026 08:51:28.585503 2337 raft_consensus.cc:2804] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:51:28.585847 2337 raft_consensus.cc:697] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 2 LEADER]: Becoming Leader. State: Replica: ab87b138f62f4b8ebb644e0f91683bbf, State: Running, Role: LEADER
I20251026 08:51:28.586123 2337 consensus_queue.cc:237] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } }
I20251026 08:51:28.587872 2024 catalog_manager.cc:5649] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf reported cstate change: term changed from 1 to 2, leader changed from dc1db092db97463194f99e71711ed461 (127.0.103.67) to ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65). New cstate: current_term: 2 leader_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:28.963577 2275 raft_consensus.cc:1275] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 2 FOLLOWER]: Refusing update from remote peer ab87b138f62f4b8ebb644e0f91683bbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:51:28.964205 2337 consensus_queue.cc:1048] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:28.967953 2201 raft_consensus.cc:1275] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 2 FOLLOWER]: Refusing update from remote peer ab87b138f62f4b8ebb644e0f91683bbf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:51:28.968546 2359 consensus_queue.cc:1048] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:30.150768 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:30.151228 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:30.151341 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:32.151628 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:32.152089 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:32.152226 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:34.152631 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:34.153151 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:34.153270 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:36.153741 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:36.154461 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:36.154620 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:38.154937 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:38.155601 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:38.155726 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:38.673482 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:38.675952 2372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:38.676002 2373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:38.676563 2375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:38.677102 413 server_base.cc:1047] running on GCE node
I20251026 08:51:38.678048 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:38.678153 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:38.678215 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468698678197 us; error 0 us; skew 500 ppm
I20251026 08:51:38.678388 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:38.679558 413 webserver.cc:492] Webserver started at http://127.0.103.68:40679/ using document root <none> and password file <none>
I20251026 08:51:38.679800 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:38.679877 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:38.680027 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:38.680619 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "b24fbc13dd7f45c1adfb73aea534331a"
format_stamp: "Formatted at 2025-10-26 08:51:38 on dist-test-slave-rd83"
I20251026 08:51:38.682809 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:51:38.684128 2380 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:38.684420 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:51:38.684506 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "b24fbc13dd7f45c1adfb73aea534331a"
format_stamp: "Formatted at 2025-10-26 08:51:38 on dist-test-slave-rd83"
I20251026 08:51:38.684587 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:38.705050 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:38.705447 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:38.705859 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:51:38.706595 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:51:38.706665 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:38.706722 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:51:38.706774 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:38.720458 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:42103
I20251026 08:51:38.720499 2442 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:42103 every 8 connection(s)
I20251026 08:51:38.726387 2443 heartbeater.cc:344] Connected to a master server at 127.0.103.126:45685
I20251026 08:51:38.726500 2443 heartbeater.cc:461] Registering TS with master...
I20251026 08:51:38.726776 2443 heartbeater.cc:507] Master 127.0.103.126:45685 requested a full tablet report, sending...
I20251026 08:51:38.727422 2024 ts_manager.cc:194] Registered new tserver with Master: b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68:42103)
I20251026 08:51:38.728058 2024 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49894
I20251026 08:51:39.163954 2275 consensus_queue.cc:237] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } }
I20251026 08:51:39.166694 2126 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:39.166913 2201 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:51:39.167301 2450 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:39.168097 2450 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:51:39.170644 2370 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } } }
I20251026 08:51:39.171263 2126 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } } }
I20251026 08:51:39.174424 2201 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } } }
W20251026 08:51:39.175328 2238 consensus_peers.cc:597] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 -> Peer b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68:42103): Couldn't send request to peer b24fbc13dd7f45c1adfb73aea534331a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f38d614def11479b80211a50c3c0ce25. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:51:39.175420 2024 catalog_manager.cc:5649] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 reported cstate change: config changed from index -1 to 2, NON_VOTER b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "dc1db092db97463194f99e71711ed461" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:51:39.642500 2460 ts_tablet_manager.cc:933] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: Initiating tablet copy from peer dc1db092db97463194f99e71711ed461 (127.0.103.67:36629)
I20251026 08:51:39.643529 2460 tablet_copy_client.cc:323] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:36629
I20251026 08:51:39.648555 2285 tablet_copy_service.cc:140] P dc1db092db97463194f99e71711ed461: Received BeginTabletCopySession request for tablet f38d614def11479b80211a50c3c0ce25 from peer b24fbc13dd7f45c1adfb73aea534331a ({username='slave'} at 127.0.0.1:34596)
I20251026 08:51:39.648700 2285 tablet_copy_service.cc:161] P dc1db092db97463194f99e71711ed461: Beginning new tablet copy session on tablet f38d614def11479b80211a50c3c0ce25 from peer b24fbc13dd7f45c1adfb73aea534331a at {username='slave'} at 127.0.0.1:34596: session id = b24fbc13dd7f45c1adfb73aea534331a-f38d614def11479b80211a50c3c0ce25
I20251026 08:51:39.651569 2285 tablet_copy_source_session.cc:215] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:51:39.653160 2460 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f38d614def11479b80211a50c3c0ce25. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:39.657790 2460 tablet_copy_client.cc:806] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: tablet copy: Starting download of 0 data blocks...
I20251026 08:51:39.658116 2460 tablet_copy_client.cc:670] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: tablet copy: Starting download of 1 WAL segments...
I20251026 08:51:39.659927 2460 tablet_copy_client.cc:538] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:51:39.662312 2460 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: Bootstrap starting.
I20251026 08:51:39.667162 2460 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:51:39.667443 2460 tablet_bootstrap.cc:492] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: Bootstrap complete.
I20251026 08:51:39.667614 2460 ts_tablet_manager.cc:1403] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:51:39.668155 2460 raft_consensus.cc:359] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } }
I20251026 08:51:39.668303 2460 raft_consensus.cc:740] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b24fbc13dd7f45c1adfb73aea534331a, State: Initialized, Role: LEARNER
I20251026 08:51:39.668479 2460 consensus_queue.cc:260] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: true } }
I20251026 08:51:39.669044 2460 ts_tablet_manager.cc:1434] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:51:39.669108 2443 heartbeater.cc:499] Master 127.0.103.126:45685 was elected leader, sending a full tablet report...
I20251026 08:51:39.670034 2285 tablet_copy_service.cc:342] P dc1db092db97463194f99e71711ed461: Request end of tablet copy session b24fbc13dd7f45c1adfb73aea534331a-f38d614def11479b80211a50c3c0ce25 received from {username='slave'} at 127.0.0.1:34596
I20251026 08:51:39.670243 2285 tablet_copy_service.cc:434] P dc1db092db97463194f99e71711ed461: ending tablet copy session b24fbc13dd7f45c1adfb73aea534331a-f38d614def11479b80211a50c3c0ce25 on tablet f38d614def11479b80211a50c3c0ce25 with peer b24fbc13dd7f45c1adfb73aea534331a
I20251026 08:51:40.156055 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:40.156664 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:40.156833 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:40.157790 2418 raft_consensus.cc:1217] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 08:51:40.548656 2466 raft_consensus.cc:1064] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: attempting to promote NON_VOTER b24fbc13dd7f45c1adfb73aea534331a to VOTER
I20251026 08:51:40.549422 2466 consensus_queue.cc:237] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } }
I20251026 08:51:40.551622 2201 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:51:40.552794 2465 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:51:40.552984 2126 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:51:40.552984 2418 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 LEARNER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:51:40.553817 2450 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:51:40.554287 2450 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:51:40.566555 2450 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.567109 2201 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.569407 2126 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.569885 2418 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.570811 2023 catalog_manager.cc:5649] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 reported cstate change: config changed from index 2 to 3, b24fbc13dd7f45c1adfb73aea534331a (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dc1db092db97463194f99e71711ed461" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab87b138f62f4b8ebb644e0f91683bbf" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 44581 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:51:40.579275 2275 consensus_queue.cc:237] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } }
I20251026 08:51:40.580803 2418 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 FOLLOWER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:51:40.581038 2201 raft_consensus.cc:1275] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Refusing update from remote peer dc1db092db97463194f99e71711ed461: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:51:40.581563 2466 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:51:40.581787 2468 consensus_queue.cc:1048] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:51:40.583427 2466 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.583879 2418 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.585987 2008 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet f38d614def11479b80211a50c3c0ce25 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:51:40.586966 2201 raft_consensus.cc:2955] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } } }
I20251026 08:51:40.587054 2023 catalog_manager.cc:5649] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 reported cstate change: config changed from index 3 to 4, VOTER ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "dc1db092db97463194f99e71711ed461" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dc1db092db97463194f99e71711ed461" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 36629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7a8b19dfcb343b0b02c7e4a0b9c84d9" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 42673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b24fbc13dd7f45c1adfb73aea534331a" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 42103 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:51:40.591267 2116 tablet_service.cc:1552] Processing DeleteTablet for tablet f38d614def11479b80211a50c3c0ce25 with delete_type TABLET_DATA_TOMBSTONED (TS ab87b138f62f4b8ebb644e0f91683bbf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51590
I20251026 08:51:40.592087 2477 tablet_replica.cc:333] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: stopping tablet replica
I20251026 08:51:40.592288 2477 raft_consensus.cc:2243] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:40.592532 2477 raft_consensus.cc:2272] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:40.593254 2477 ts_tablet_manager.cc:1916] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:51:40.596848 2477 ts_tablet_manager.cc:1929] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 08:51:40.596975 2477 log.cc:1199] T f38d614def11479b80211a50c3c0ce25 P ab87b138f62f4b8ebb644e0f91683bbf: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1761468669168814-413-0/minicluster-data/ts-0-root/wals/f38d614def11479b80211a50c3c0ce25
I20251026 08:51:40.597561 2008 catalog_manager.cc:4985] TS ab87b138f62f4b8ebb644e0f91683bbf (127.0.103.65:44581): tablet f38d614def11479b80211a50c3c0ce25 (table test-workload [id=03abb4fea69b46e986dbca1ae2d9e01e]) successfully deleted
I20251026 08:51:42.157140 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:42.157595 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:42.157708 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:44.158103 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:44.158715 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:44.158820 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:46.159161 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:46.159664 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:46.159742 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:48.160195 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:48.161180 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:48.161347 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:50.161742 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:50.162330 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:50.162418 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:52.162734 2077 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:51:52.163221 2077 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:51:52.163311 2077 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:51:52.802556 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:51:52.812893 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:52.813143 413 tablet_replica.cc:333] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf: stopping tablet replica
I20251026 08:51:52.813316 413 raft_consensus.cc:2243] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:51:52.813582 413 raft_consensus.cc:2272] T ad2c20814a3946d2a51700e7541ccbce P ab87b138f62f4b8ebb644e0f91683bbf [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.827381 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:51:52.829979 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:51:52.837606 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:52.837865 413 tablet_replica.cc:333] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9: stopping tablet replica
I20251026 08:51:52.838017 413 raft_consensus.cc:2243] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:52.838299 413 raft_consensus.cc:2272] T f38d614def11479b80211a50c3c0ce25 P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.838827 413 tablet_replica.cc:333] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9: stopping tablet replica
I20251026 08:51:52.838975 413 raft_consensus.cc:2243] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:52.839130 413 raft_consensus.cc:2272] T ad2c20814a3946d2a51700e7541ccbce P a7a8b19dfcb343b0b02c7e4a0b9c84d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.852100 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:51:52.854470 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:51:52.862543 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:52.862744 413 tablet_replica.cc:333] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461: stopping tablet replica
I20251026 08:51:52.862870 413 raft_consensus.cc:2243] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:52.863111 413 raft_consensus.cc:2272] T f38d614def11479b80211a50c3c0ce25 P dc1db092db97463194f99e71711ed461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.863505 413 tablet_replica.cc:333] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461: stopping tablet replica
I20251026 08:51:52.863623 413 raft_consensus.cc:2243] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:52.863745 413 raft_consensus.cc:2272] T ad2c20814a3946d2a51700e7541ccbce P dc1db092db97463194f99e71711ed461 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.878412 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:51:52.880959 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:51:52.889470 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:51:52.889707 413 tablet_replica.cc:333] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a: stopping tablet replica
I20251026 08:51:52.889861 413 raft_consensus.cc:2243] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:51:52.890106 413 raft_consensus.cc:2272] T f38d614def11479b80211a50c3c0ce25 P b24fbc13dd7f45c1adfb73aea534331a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.902822 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:51:52.909794 413 master.cc:561] Master@127.0.103.126:45685 shutting down...
I20251026 08:51:52.916867 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:51:52.917009 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:51:52.917091 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 69860859464d44de8576fe622aec1b9e: stopping tablet replica
I20251026 08:51:52.930570 413 master.cc:583] Master@127.0.103.126:45685 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (26863 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20251026 08:51:52.940747 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36649
I20251026 08:51:52.941223 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:51:52.943457 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:52.943495 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:51:52.943586 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:51:52.943930 413 server_base.cc:1047] running on GCE node
I20251026 08:51:52.944078 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:51:52.944123 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:51:52.944142 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468712944140 us; error 0 us; skew 500 ppm
I20251026 08:51:52.944238 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:51:52.945250 413 webserver.cc:492] Webserver started at http://127.0.103.126:35673/ using document root <none> and password file <none>
I20251026 08:51:52.945441 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:51:52.945482 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:51:52.945554 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:51:52.946039 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "85b64630dd7a44929a08a5031bc353ba"
format_stamp: "Formatted at 2025-10-26 08:51:52 on dist-test-slave-rd83"
I20251026 08:51:52.948132 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:51:52.949394 2491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:51:52.949791 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:51:52.949875 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "85b64630dd7a44929a08a5031bc353ba"
format_stamp: "Formatted at 2025-10-26 08:51:52 on dist-test-slave-rd83"
I20251026 08:51:52.949988 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:51:52.960608 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:51:52.961023 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:51:52.970472 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36649
I20251026 08:51:52.970487 2542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36649 every 8 connection(s)
I20251026 08:51:52.971478 2543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:51:52.974813 2543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba: Bootstrap starting.
I20251026 08:51:52.975687 2543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba: Neither blocks nor log segments found. Creating new log.
I20251026 08:51:52.976810 2543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba: No bootstrap required, opened a new log
I20251026 08:51:52.977301 2543 raft_consensus.cc:359] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b64630dd7a44929a08a5031bc353ba" member_type: VOTER }
I20251026 08:51:52.977401 2543 raft_consensus.cc:385] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:51:52.977448 2543 raft_consensus.cc:740] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b64630dd7a44929a08a5031bc353ba, State: Initialized, Role: FOLLOWER
I20251026 08:51:52.977609 2543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b64630dd7a44929a08a5031bc353ba" member_type: VOTER }
I20251026 08:51:52.977695 2543 raft_consensus.cc:399] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:51:52.977748 2543 raft_consensus.cc:493] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:51:52.977824 2543 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:51:52.979393 2543 raft_consensus.cc:515] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b64630dd7a44929a08a5031bc353ba" member_type: VOTER }
I20251026 08:51:52.979542 2543 leader_election.cc:304] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85b64630dd7a44929a08a5031bc353ba; no voters:
I20251026 08:51:52.979893 2543 leader_election.cc:290] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:51:52.979987 2546 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:51:52.980415 2546 raft_consensus.cc:697] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 1 LEADER]: Becoming Leader. State: Replica: 85b64630dd7a44929a08a5031bc353ba, State: Running, Role: LEADER
I20251026 08:51:52.980612 2543 sys_catalog.cc:565] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:51:52.980607 2546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b64630dd7a44929a08a5031bc353ba" member_type: VOTER }
I20251026 08:51:52.981370 2548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85b64630dd7a44929a08a5031bc353ba. Latest consensus state: current_term: 1 leader_uuid: "85b64630dd7a44929a08a5031bc353ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b64630dd7a44929a08a5031bc353ba" member_type: VOTER } }
I20251026 08:51:52.981587 2548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:52.981774 2547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85b64630dd7a44929a08a5031bc353ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85b64630dd7a44929a08a5031bc353ba" member_type: VOTER } }
I20251026 08:51:52.982017 2547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [sys.catalog]: This master's current role is: LEADER
I20251026 08:51:52.985759 2551 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:51:52.985956 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:51:52.986884 2551 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:51:52.989176 2551 catalog_manager.cc:1357] Generated new cluster ID: c9ba6ae9497942db8b2ff56a02c4c4d1
I20251026 08:51:52.989244 2551 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:51:53.029458 2551 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:51:53.030093 2551 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:51:53.039593 2551 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba: Generated new TSK 0
I20251026 08:51:53.039732 2551 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:51:53.050019 413 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 4.6009e-05s
I20251026 08:52:05.131227 413 master.cc:561] Master@127.0.103.126:36649 shutting down...
I20251026 08:52:05.143709 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:52:05.143903 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:05.143976 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 85b64630dd7a44929a08a5031bc353ba: stopping tablet replica
I20251026 08:52:05.157269 413 master.cc:583] Master@127.0.103.126:36649 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12223 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20251026 08:52:05.164083 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:44003
I20251026 08:52:05.164467 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:05.166659 2565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.166697 2566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.166847 2568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:05.168757 413 server_base.cc:1047] running on GCE node
I20251026 08:52:05.170424 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:05.170490 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:05.170521 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468725170509 us; error 0 us; skew 500 ppm
I20251026 08:52:05.170622 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:05.171572 413 webserver.cc:492] Webserver started at http://127.0.103.126:45863/ using document root <none> and password file <none>
I20251026 08:52:05.171736 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:05.171787 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:05.171880 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:05.172392 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "32c4ebf378a5433794eada568fce3394"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.174571 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:05.175813 2573 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.176123 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:05.176239 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "32c4ebf378a5433794eada568fce3394"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.176330 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:05.187214 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:05.187638 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:05.196974 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:44003
I20251026 08:52:05.196993 2624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:44003 every 8 connection(s)
I20251026 08:52:05.197997 2625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:05.201602 2625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394: Bootstrap starting.
I20251026 08:52:05.202675 2625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:05.203938 2625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394: No bootstrap required, opened a new log
I20251026 08:52:05.204424 2625 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4ebf378a5433794eada568fce3394" member_type: VOTER }
I20251026 08:52:05.204543 2625 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:05.204588 2625 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c4ebf378a5433794eada568fce3394, State: Initialized, Role: FOLLOWER
I20251026 08:52:05.204780 2625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4ebf378a5433794eada568fce3394" member_type: VOTER }
I20251026 08:52:05.204881 2625 raft_consensus.cc:399] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:52:05.204942 2625 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:52:05.205027 2625 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:05.206593 2625 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4ebf378a5433794eada568fce3394" member_type: VOTER }
I20251026 08:52:05.206746 2625 leader_election.cc:304] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c4ebf378a5433794eada568fce3394; no voters:
I20251026 08:52:05.207070 2625 leader_election.cc:290] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:52:05.207176 2628 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:05.207629 2628 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 1 LEADER]: Becoming Leader. State: Replica: 32c4ebf378a5433794eada568fce3394, State: Running, Role: LEADER
I20251026 08:52:05.207815 2625 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:52:05.207823 2628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4ebf378a5433794eada568fce3394" member_type: VOTER }
I20251026 08:52:05.208568 2630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c4ebf378a5433794eada568fce3394. Latest consensus state: current_term: 1 leader_uuid: "32c4ebf378a5433794eada568fce3394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4ebf378a5433794eada568fce3394" member_type: VOTER } }
I20251026 08:52:05.208714 2630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:05.208632 2629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c4ebf378a5433794eada568fce3394" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c4ebf378a5433794eada568fce3394" member_type: VOTER } }
I20251026 08:52:05.208814 2629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:05.209798 2633 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:52:05.211485 2633 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:52:05.212461 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:52:05.214511 2633 catalog_manager.cc:1357] Generated new cluster ID: 543b46a96f0b4b268abba8bf137e96e8
I20251026 08:52:05.214602 2633 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:52:05.230161 2633 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:52:05.230777 2633 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:52:05.237614 2633 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394: Generated new TSK 0
I20251026 08:52:05.237793 2633 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:52:05.245188 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:05.247300 2646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.248037 2647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.248078 2649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:05.248391 413 server_base.cc:1047] running on GCE node
I20251026 08:52:05.248601 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:05.248656 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:05.248708 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468725248695 us; error 0 us; skew 500 ppm
I20251026 08:52:05.248807 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:05.249780 413 webserver.cc:492] Webserver started at http://127.0.103.65:36755/ using document root <none> and password file <none>
I20251026 08:52:05.249970 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:05.250034 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:05.250166 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:05.250659 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "4dfa59ea0d9941ee82c55e402f9d1dbc"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.252776 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:05.254130 2654 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.254465 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:52:05.254549 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "4dfa59ea0d9941ee82c55e402f9d1dbc"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.254653 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:05.266602 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:05.267009 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:05.267433 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:05.268251 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:05.268316 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.268395 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:05.268441 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.281208 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:45469
I20251026 08:52:05.281237 2716 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:45469 every 8 connection(s)
I20251026 08:52:05.282629 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:05.285231 2722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.285323 2721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.286770 2724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:05.286867 413 server_base.cc:1047] running on GCE node
I20251026 08:52:05.287257 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:05.287329 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:05.287388 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468725287367 us; error 0 us; skew 500 ppm
I20251026 08:52:05.287529 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:05.288524 413 webserver.cc:492] Webserver started at http://127.0.103.66:40479/ using document root <none> and password file <none>
I20251026 08:52:05.288708 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:05.288774 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:05.288887 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:05.289397 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "3c8e935812d847e38a46572c7089a189"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.291539 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:05.291831 2717 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44003
I20251026 08:52:05.291953 2717 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:05.292439 2717 heartbeater.cc:507] Master 127.0.103.126:44003 requested a full tablet report, sending...
I20251026 08:52:05.293200 2729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.293502 2590 ts_manager.cc:194] Registered new tserver with Master: 4dfa59ea0d9941ee82c55e402f9d1dbc (127.0.103.65:45469)
I20251026 08:52:05.293581 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251026 08:52:05.293682 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "3c8e935812d847e38a46572c7089a189"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.293779 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:05.294657 2590 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36460
I20251026 08:52:05.315164 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:05.315531 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:05.316018 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:05.316651 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:05.316701 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.316763 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:05.316797 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.328578 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:34087
I20251026 08:52:05.328658 2791 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:34087 every 8 connection(s)
I20251026 08:52:05.329916 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:05.332374 2795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.332942 2796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:05.334040 2798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:05.335834 2792 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44003
I20251026 08:52:05.335952 2792 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:05.336203 2792 heartbeater.cc:507] Master 127.0.103.126:44003 requested a full tablet report, sending...
I20251026 08:52:05.336916 2590 ts_manager.cc:194] Registered new tserver with Master: 3c8e935812d847e38a46572c7089a189 (127.0.103.66:34087)
I20251026 08:52:05.338208 2590 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36468
I20251026 08:52:05.339646 413 server_base.cc:1047] running on GCE node
I20251026 08:52:05.339957 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:05.340029 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:05.340070 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468725340065 us; error 0 us; skew 500 ppm
I20251026 08:52:05.340211 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:05.341218 413 webserver.cc:492] Webserver started at http://127.0.103.67:42711/ using document root <none> and password file <none>
I20251026 08:52:05.341396 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:05.341459 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:05.341570 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:05.342131 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "92f5432a2d284a5e9ac942b9aec8b363"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.344542 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251026 08:52:05.345875 2803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.346182 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:52:05.346300 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "92f5432a2d284a5e9ac942b9aec8b363"
format_stamp: "Formatted at 2025-10-26 08:52:05 on dist-test-slave-rd83"
I20251026 08:52:05.346391 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:05.366001 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:05.366451 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:05.366843 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:05.367440 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:05.367486 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.367543 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:05.367579 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:05.379577 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:42415
I20251026 08:52:05.379609 2865 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:42415 every 8 connection(s)
I20251026 08:52:05.384840 2866 heartbeater.cc:344] Connected to a master server at 127.0.103.126:44003
I20251026 08:52:05.384927 2866 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:05.385164 2866 heartbeater.cc:507] Master 127.0.103.126:44003 requested a full tablet report, sending...
I20251026 08:52:05.385818 2590 ts_manager.cc:194] Registered new tserver with Master: 92f5432a2d284a5e9ac942b9aec8b363 (127.0.103.67:42415)
I20251026 08:52:05.385943 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005615502s
I20251026 08:52:05.386794 2590 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:36480
I20251026 08:52:06.295892 2717 heartbeater.cc:499] Master 127.0.103.126:44003 was elected leader, sending a full tablet report...
I20251026 08:52:06.339684 2792 heartbeater.cc:499] Master 127.0.103.126:44003 was elected leader, sending a full tablet report...
I20251026 08:52:06.387766 2866 heartbeater.cc:499] Master 127.0.103.126:44003 was elected leader, sending a full tablet report...
I20251026 08:52:07.212334 2643 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:09.212767 2643 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:11.213146 2643 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:13.213524 2643 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:15.213941 2643 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:17.214356 2643 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:17.471213 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:52:17.483299 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:17.497256 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:52:17.499603 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:52:17.507040 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:17.519075 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:52:17.521168 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:52:17.528052 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:17.541643 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:52:17.543707 413 master.cc:561] Master@127.0.103.126:44003 shutting down...
I20251026 08:52:17.550671 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:52:17.550858 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:17.550956 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 32c4ebf378a5433794eada568fce3394: stopping tablet replica
I20251026 08:52:17.563361 413 master.cc:583] Master@127.0.103.126:44003 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12408 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20251026 08:52:17.572335 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:43477
I20251026 08:52:17.572765 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:17.574965 2871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.574975 2870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.575294 2873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:17.575446 413 server_base.cc:1047] running on GCE node
I20251026 08:52:17.575752 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:17.575814 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:17.575850 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468737575837 us; error 0 us; skew 500 ppm
I20251026 08:52:17.575956 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:17.576970 413 webserver.cc:492] Webserver started at http://127.0.103.126:39943/ using document root <none> and password file <none>
I20251026 08:52:17.577152 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:17.577211 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:17.577314 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:17.577833 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "23561ebb5eea485b8c659f5deeaec36f"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.580116 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:52:17.581307 2878 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.581579 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:17.581674 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "23561ebb5eea485b8c659f5deeaec36f"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.581766 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:17.593114 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:17.593612 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:17.604293 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:43477
I20251026 08:52:17.604326 2929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:43477 every 8 connection(s)
I20251026 08:52:17.605409 2930 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:17.608783 2930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f: Bootstrap starting.
I20251026 08:52:17.609853 2930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:17.611192 2930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f: No bootstrap required, opened a new log
I20251026 08:52:17.611704 2930 raft_consensus.cc:359] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23561ebb5eea485b8c659f5deeaec36f" member_type: VOTER }
I20251026 08:52:17.611831 2930 raft_consensus.cc:385] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:17.611917 2930 raft_consensus.cc:740] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23561ebb5eea485b8c659f5deeaec36f, State: Initialized, Role: FOLLOWER
I20251026 08:52:17.612100 2930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23561ebb5eea485b8c659f5deeaec36f" member_type: VOTER }
I20251026 08:52:17.612206 2930 raft_consensus.cc:399] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:52:17.612277 2930 raft_consensus.cc:493] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:52:17.612362 2930 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:17.613832 2930 raft_consensus.cc:515] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23561ebb5eea485b8c659f5deeaec36f" member_type: VOTER }
I20251026 08:52:17.614006 2930 leader_election.cc:304] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23561ebb5eea485b8c659f5deeaec36f; no voters:
I20251026 08:52:17.614387 2930 leader_election.cc:290] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:52:17.614482 2933 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:17.614979 2933 raft_consensus.cc:697] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 1 LEADER]: Becoming Leader. State: Replica: 23561ebb5eea485b8c659f5deeaec36f, State: Running, Role: LEADER
I20251026 08:52:17.615201 2930 sys_catalog.cc:565] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:52:17.615204 2933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23561ebb5eea485b8c659f5deeaec36f" member_type: VOTER }
I20251026 08:52:17.616110 2934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23561ebb5eea485b8c659f5deeaec36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23561ebb5eea485b8c659f5deeaec36f" member_type: VOTER } }
I20251026 08:52:17.616163 2935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23561ebb5eea485b8c659f5deeaec36f. Latest consensus state: current_term: 1 leader_uuid: "23561ebb5eea485b8c659f5deeaec36f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23561ebb5eea485b8c659f5deeaec36f" member_type: VOTER } }
I20251026 08:52:17.616374 2934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:17.616384 2935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:17.618702 2938 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:52:17.620082 2938 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:52:17.620438 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:52:17.622453 2938 catalog_manager.cc:1357] Generated new cluster ID: a642496dadc240afa7f25fc9a42b54f8
I20251026 08:52:17.622517 2938 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:52:17.632539 2938 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:52:17.633142 2938 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:52:17.641806 2938 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f: Generated new TSK 0
I20251026 08:52:17.641958 2938 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:52:17.652957 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:17.655210 2951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.655481 2952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.655933 2954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:17.656136 413 server_base.cc:1047] running on GCE node
I20251026 08:52:17.656358 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:17.656420 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:17.656481 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468737656460 us; error 0 us; skew 500 ppm
I20251026 08:52:17.656631 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:17.657608 413 webserver.cc:492] Webserver started at http://127.0.103.65:36955/ using document root <none> and password file <none>
I20251026 08:52:17.657812 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:17.657878 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:17.658030 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:17.658918 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "9ad36d6fa670457287fc0614c9cb1d60"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.662111 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20251026 08:52:17.664049 2959 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.664472 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251026 08:52:17.664592 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "9ad36d6fa670457287fc0614c9cb1d60"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.664707 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:17.674516 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:17.675034 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:17.675510 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:17.676230 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:17.676295 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.676412 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:17.676468 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.688524 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:37739
I20251026 08:52:17.688544 3021 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:37739 every 8 connection(s)
I20251026 08:52:17.689924 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:17.692626 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.692734 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.693826 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:17.695066 413 server_base.cc:1047] running on GCE node
I20251026 08:52:17.695303 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:17.695376 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:17.695467 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468737695415 us; error 0 us; skew 500 ppm
I20251026 08:52:17.695622 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:17.697073 413 webserver.cc:492] Webserver started at http://127.0.103.66:36277/ using document root <none> and password file <none>
I20251026 08:52:17.697327 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:17.697403 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:17.697544 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:17.697594 3022 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43477
I20251026 08:52:17.697741 3022 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:17.698024 3022 heartbeater.cc:507] Master 127.0.103.126:43477 requested a full tablet report, sending...
I20251026 08:52:17.698175 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "54252cf7077544a49fdd5601e9f1e44a"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.698879 2895 ts_manager.cc:194] Registered new tserver with Master: 9ad36d6fa670457287fc0614c9cb1d60 (127.0.103.65:37739)
I20251026 08:52:17.699891 2895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48088
I20251026 08:52:17.700645 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:17.702134 3034 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.702431 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:17.702531 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "54252cf7077544a49fdd5601e9f1e44a"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.702621 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:17.719928 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:17.720291 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:17.720714 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:17.721356 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:17.721407 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.721469 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:17.721505 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.730669 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:45411
I20251026 08:52:17.730690 3096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:45411 every 8 connection(s)
I20251026 08:52:17.731886 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:17.734269 3101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.734519 3100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.735468 3103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:17.736361 413 server_base.cc:1047] running on GCE node
I20251026 08:52:17.736533 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:17.736585 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:17.736606 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468737736603 us; error 0 us; skew 500 ppm
I20251026 08:52:17.736735 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:17.736812 3097 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43477
I20251026 08:52:17.736925 3097 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:17.737212 3097 heartbeater.cc:507] Master 127.0.103.126:43477 requested a full tablet report, sending...
I20251026 08:52:17.737761 413 webserver.cc:492] Webserver started at http://127.0.103.67:39391/ using document root <none> and password file <none>
I20251026 08:52:17.737931 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:17.737987 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:17.737994 2895 ts_manager.cc:194] Registered new tserver with Master: 54252cf7077544a49fdd5601e9f1e44a (127.0.103.66:45411)
I20251026 08:52:17.738123 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:17.738659 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "7b88d94dd38944e18bd7c74c4282b3e8"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.738754 2895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48096
I20251026 08:52:17.740733 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251026 08:52:17.741863 3108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.742161 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:52:17.742239 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "7b88d94dd38944e18bd7c74c4282b3e8"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.742318 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:17.750216 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:17.750564 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:17.750929 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:17.751515 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:17.751560 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.751614 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:17.751645 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.761029 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:41137
I20251026 08:52:17.761051 3170 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:41137 every 8 connection(s)
I20251026 08:52:17.762283 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:17.764349 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.764453 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:17.764961 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:17.767155 413 server_base.cc:1047] running on GCE node
I20251026 08:52:17.767417 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:17.767484 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:17.767519 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468737767516 us; error 0 us; skew 500 ppm
I20251026 08:52:17.767652 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:17.768177 3171 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43477
I20251026 08:52:17.768275 3171 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:17.768501 3171 heartbeater.cc:507] Master 127.0.103.126:43477 requested a full tablet report, sending...
I20251026 08:52:17.768723 413 webserver.cc:492] Webserver started at http://127.0.103.68:36781/ using document root <none> and password file <none>
I20251026 08:52:17.768886 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:17.768939 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:17.769034 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:17.769170 2895 ts_manager.cc:194] Registered new tserver with Master: 7b88d94dd38944e18bd7c74c4282b3e8 (127.0.103.67:41137)
I20251026 08:52:17.769528 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "c3da039e43fe40b68b180e05c096e00b"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.769856 2895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48104
I20251026 08:52:17.771469 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:17.772681 3182 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.772984 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:17.773051 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "c3da039e43fe40b68b180e05c096e00b"
format_stamp: "Formatted at 2025-10-26 08:52:17 on dist-test-slave-rd83"
I20251026 08:52:17.773113 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:17.797309 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:17.797613 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:17.797986 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:17.798660 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:17.798710 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.798750 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:17.798779 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:17.809139 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:37523
I20251026 08:52:17.809165 3244 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:37523 every 8 connection(s)
I20251026 08:52:17.814833 3245 heartbeater.cc:344] Connected to a master server at 127.0.103.126:43477
I20251026 08:52:17.814936 3245 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:17.815209 3245 heartbeater.cc:507] Master 127.0.103.126:43477 requested a full tablet report, sending...
I20251026 08:52:17.815852 2895 ts_manager.cc:194] Registered new tserver with Master: c3da039e43fe40b68b180e05c096e00b (127.0.103.68:37523)
I20251026 08:52:17.816524 2895 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48120
I20251026 08:52:17.816629 413 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.006742994s
I20251026 08:52:18.701256 3022 heartbeater.cc:499] Master 127.0.103.126:43477 was elected leader, sending a full tablet report...
I20251026 08:52:18.739796 3097 heartbeater.cc:499] Master 127.0.103.126:43477 was elected leader, sending a full tablet report...
I20251026 08:52:18.770869 3171 heartbeater.cc:499] Master 127.0.103.126:43477 was elected leader, sending a full tablet report...
I20251026 08:52:18.817730 3245 heartbeater.cc:499] Master 127.0.103.126:43477 was elected leader, sending a full tablet report...
I20251026 08:52:18.850008 413 test_util.cc:276] Using random seed: -428454700
I20251026 08:52:18.859714 2895 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48134:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251026 08:52:18.874461 3210 tablet_service.cc:1505] Processing CreateTablet for tablet f9506a8812364038b36bbd8465403885 (DEFAULT_TABLE table=test-workload [id=773632959232491cb18ca996056ec2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:52:18.874611 3136 tablet_service.cc:1505] Processing CreateTablet for tablet f9506a8812364038b36bbd8465403885 (DEFAULT_TABLE table=test-workload [id=773632959232491cb18ca996056ec2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:52:18.874876 3210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9506a8812364038b36bbd8465403885. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:18.875041 3136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9506a8812364038b36bbd8465403885. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:18.879055 3266 tablet_bootstrap.cc:492] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8: Bootstrap starting.
I20251026 08:52:18.879055 2987 tablet_service.cc:1505] Processing CreateTablet for tablet f9506a8812364038b36bbd8465403885 (DEFAULT_TABLE table=test-workload [id=773632959232491cb18ca996056ec2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:52:18.879115 3267 tablet_bootstrap.cc:492] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b: Bootstrap starting.
I20251026 08:52:18.879454 2987 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9506a8812364038b36bbd8465403885. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:18.880844 3267 tablet_bootstrap.cc:654] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:18.880854 3266 tablet_bootstrap.cc:654] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:18.882579 3267 tablet_bootstrap.cc:492] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b: No bootstrap required, opened a new log
I20251026 08:52:18.882685 3267 ts_tablet_manager.cc:1403] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:52:18.882901 3266 tablet_bootstrap.cc:492] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8: No bootstrap required, opened a new log
I20251026 08:52:18.883039 3266 ts_tablet_manager.cc:1403] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20251026 08:52:18.883369 3267 raft_consensus.cc:359] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:18.883522 3267 raft_consensus.cc:385] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:18.883612 3267 raft_consensus.cc:740] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3da039e43fe40b68b180e05c096e00b, State: Initialized, Role: FOLLOWER
I20251026 08:52:18.883566 3266 raft_consensus.cc:359] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:18.883709 3266 raft_consensus.cc:385] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:18.883790 3266 raft_consensus.cc:740] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b88d94dd38944e18bd7c74c4282b3e8, State: Initialized, Role: FOLLOWER
I20251026 08:52:18.883875 3267 consensus_queue.cc:260] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:18.884020 3266 consensus_queue.cc:260] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:18.884347 3270 tablet_bootstrap.cc:492] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: Bootstrap starting.
I20251026 08:52:18.885046 3267 ts_tablet_manager.cc:1434] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:18.885143 3266 ts_tablet_manager.cc:1434] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:52:18.886544 3270 tablet_bootstrap.cc:654] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:18.888955 3270 tablet_bootstrap.cc:492] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: No bootstrap required, opened a new log
I20251026 08:52:18.889037 3270 ts_tablet_manager.cc:1403] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251026 08:52:18.889438 3270 raft_consensus.cc:359] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:18.889540 3270 raft_consensus.cc:385] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:18.889572 3270 raft_consensus.cc:740] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ad36d6fa670457287fc0614c9cb1d60, State: Initialized, Role: FOLLOWER
I20251026 08:52:18.889680 3270 consensus_queue.cc:260] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:18.890152 3270 ts_tablet_manager.cc:1434] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20251026 08:52:18.940392 3023 tablet.cc:2378] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 08:52:19.121218 3274 raft_consensus.cc:493] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:19.121384 3274 raft_consensus.cc:515] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:19.122174 3274 leader_election.cc:290] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3da039e43fe40b68b180e05c096e00b (127.0.103.68:37523), 7b88d94dd38944e18bd7c74c4282b3e8 (127.0.103.67:41137)
I20251026 08:52:19.128674 3220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9506a8812364038b36bbd8465403885" candidate_uuid: "9ad36d6fa670457287fc0614c9cb1d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3da039e43fe40b68b180e05c096e00b" is_pre_election: true
I20251026 08:52:19.128759 3146 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9506a8812364038b36bbd8465403885" candidate_uuid: "9ad36d6fa670457287fc0614c9cb1d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" is_pre_election: true
I20251026 08:52:19.128856 3220 raft_consensus.cc:2468] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad36d6fa670457287fc0614c9cb1d60 in term 0.
I20251026 08:52:19.128909 3146 raft_consensus.cc:2468] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ad36d6fa670457287fc0614c9cb1d60 in term 0.
I20251026 08:52:19.129200 2963 leader_election.cc:304] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ad36d6fa670457287fc0614c9cb1d60, c3da039e43fe40b68b180e05c096e00b; no voters:
I20251026 08:52:19.129405 3274 raft_consensus.cc:2804] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:19.129470 3274 raft_consensus.cc:493] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:19.129532 3274 raft_consensus.cc:3060] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:19.131282 3274 raft_consensus.cc:515] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:19.131758 3274 leader_election.cc:290] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [CANDIDATE]: Term 1 election: Requested vote from peers c3da039e43fe40b68b180e05c096e00b (127.0.103.68:37523), 7b88d94dd38944e18bd7c74c4282b3e8 (127.0.103.67:41137)
I20251026 08:52:19.132112 3220 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9506a8812364038b36bbd8465403885" candidate_uuid: "9ad36d6fa670457287fc0614c9cb1d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3da039e43fe40b68b180e05c096e00b"
I20251026 08:52:19.132112 3146 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f9506a8812364038b36bbd8465403885" candidate_uuid: "9ad36d6fa670457287fc0614c9cb1d60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b88d94dd38944e18bd7c74c4282b3e8"
I20251026 08:52:19.132274 3220 raft_consensus.cc:3060] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:19.132274 3146 raft_consensus.cc:3060] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:19.134866 3146 raft_consensus.cc:2468] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad36d6fa670457287fc0614c9cb1d60 in term 1.
I20251026 08:52:19.134866 3220 raft_consensus.cc:2468] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad36d6fa670457287fc0614c9cb1d60 in term 1.
I20251026 08:52:19.135223 2963 leader_election.cc:304] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7b88d94dd38944e18bd7c74c4282b3e8, 9ad36d6fa670457287fc0614c9cb1d60, c3da039e43fe40b68b180e05c096e00b; no voters:
I20251026 08:52:19.135426 3274 raft_consensus.cc:2804] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:19.135752 3274 raft_consensus.cc:697] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 1 LEADER]: Becoming Leader. State: Replica: 9ad36d6fa670457287fc0614c9cb1d60, State: Running, Role: LEADER
I20251026 08:52:19.136132 3274 consensus_queue.cc:237] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } }
I20251026 08:52:19.138502 2895 catalog_manager.cc:5649] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ad36d6fa670457287fc0614c9cb1d60 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "9ad36d6fa670457287fc0614c9cb1d60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad36d6fa670457287fc0614c9cb1d60" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 37739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:19.585925 3274 consensus_queue.cc:1048] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b88d94dd38944e18bd7c74c4282b3e8" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:19.590117 3277 consensus_queue.cc:1048] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3da039e43fe40b68b180e05c096e00b" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:19.620149 2948 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:19.620572 2948 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:19.620668 2948 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:21.620942 2948 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:21.621266 2948 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:21.621333 2948 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:23.621587 2948 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:23.621953 2948 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:23.622023 2948 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:25.622290 2948 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:25.622633 2948 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:25.622732 2948 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:27.622998 2948 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:27.623325 2948 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:27.623394 2948 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:29.623672 2948 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:29.624061 2948 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:29.624131 2948 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:30.219753 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:52:30.231875 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:30.232154 413 tablet_replica.cc:333] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60: stopping tablet replica
I20251026 08:52:30.232316 413 raft_consensus.cc:2243] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:52:30.232596 413 raft_consensus.cc:2272] T f9506a8812364038b36bbd8465403885 P 9ad36d6fa670457287fc0614c9cb1d60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:30.246417 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:52:30.249177 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:52:30.257282 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:30.270562 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:52:30.272785 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:52:30.281165 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:30.281379 413 tablet_replica.cc:333] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8: stopping tablet replica
I20251026 08:52:30.281528 413 raft_consensus.cc:2243] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:30.281697 413 raft_consensus.cc:2272] T f9506a8812364038b36bbd8465403885 P 7b88d94dd38944e18bd7c74c4282b3e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:30.294663 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:52:30.297086 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:52:30.312825 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:30.313027 413 tablet_replica.cc:333] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b: stopping tablet replica
I20251026 08:52:30.313184 413 raft_consensus.cc:2243] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:30.313328 413 raft_consensus.cc:2272] T f9506a8812364038b36bbd8465403885 P c3da039e43fe40b68b180e05c096e00b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:30.326377 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:52:30.328647 413 master.cc:561] Master@127.0.103.126:43477 shutting down...
I20251026 08:52:30.336309 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:52:30.336472 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:30.336556 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 23561ebb5eea485b8c659f5deeaec36f: stopping tablet replica
I20251026 08:52:30.349443 413 master.cc:583] Master@127.0.103.126:43477 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12787 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20251026 08:52:30.359656 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36245
I20251026 08:52:30.360083 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:30.361938 3290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:30.361941 3289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:30.362346 3292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:30.362363 413 server_base.cc:1047] running on GCE node
I20251026 08:52:30.362620 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:30.362668 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:30.362689 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468750362686 us; error 0 us; skew 500 ppm
I20251026 08:52:30.362828 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:30.363762 413 webserver.cc:492] Webserver started at http://127.0.103.126:38175/ using document root <none> and password file <none>
I20251026 08:52:30.363933 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:30.363981 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:30.364076 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:30.364594 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "518eb071f47b44429fc2c70bdb8c14f3"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.366876 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:30.368203 3297 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.368610 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:30.368741 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "518eb071f47b44429fc2c70bdb8c14f3"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.368860 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:30.386982 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:30.387394 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:30.397073 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36245
I20251026 08:52:30.397092 3348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36245 every 8 connection(s)
I20251026 08:52:30.398236 3349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:30.401810 3349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3: Bootstrap starting.
I20251026 08:52:30.402854 3349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:30.404031 3349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3: No bootstrap required, opened a new log
I20251026 08:52:30.404529 3349 raft_consensus.cc:359] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518eb071f47b44429fc2c70bdb8c14f3" member_type: VOTER }
I20251026 08:52:30.404632 3349 raft_consensus.cc:385] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:30.404682 3349 raft_consensus.cc:740] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518eb071f47b44429fc2c70bdb8c14f3, State: Initialized, Role: FOLLOWER
I20251026 08:52:30.404867 3349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518eb071f47b44429fc2c70bdb8c14f3" member_type: VOTER }
I20251026 08:52:30.404963 3349 raft_consensus.cc:399] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:52:30.405018 3349 raft_consensus.cc:493] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:52:30.405102 3349 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:30.407132 3349 raft_consensus.cc:515] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518eb071f47b44429fc2c70bdb8c14f3" member_type: VOTER }
I20251026 08:52:30.407269 3349 leader_election.cc:304] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 518eb071f47b44429fc2c70bdb8c14f3; no voters:
I20251026 08:52:30.407568 3349 leader_election.cc:290] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:52:30.407647 3352 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:30.408035 3352 raft_consensus.cc:697] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 1 LEADER]: Becoming Leader. State: Replica: 518eb071f47b44429fc2c70bdb8c14f3, State: Running, Role: LEADER
I20251026 08:52:30.408232 3352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518eb071f47b44429fc2c70bdb8c14f3" member_type: VOTER }
I20251026 08:52:30.408280 3349 sys_catalog.cc:565] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:52:30.408774 3353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "518eb071f47b44429fc2c70bdb8c14f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518eb071f47b44429fc2c70bdb8c14f3" member_type: VOTER } }
I20251026 08:52:30.408794 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 518eb071f47b44429fc2c70bdb8c14f3. Latest consensus state: current_term: 1 leader_uuid: "518eb071f47b44429fc2c70bdb8c14f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "518eb071f47b44429fc2c70bdb8c14f3" member_type: VOTER } }
I20251026 08:52:30.408958 3353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:30.409036 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:30.410144 3357 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:52:30.411496 3357 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:52:30.412708 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:52:30.414009 3357 catalog_manager.cc:1357] Generated new cluster ID: 326e8387045e4eef8e29b7e7dea9130f
I20251026 08:52:30.414091 3357 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:52:30.425077 3357 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:52:30.425727 3357 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:52:30.441123 3357 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3: Generated new TSK 0
I20251026 08:52:30.441272 3357 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:52:30.444999 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:30.447199 3370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:30.447477 3371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:30.447819 413 server_base.cc:1047] running on GCE node
W20251026 08:52:30.448241 3373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:30.448496 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:30.448549 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:30.448570 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468750448568 us; error 0 us; skew 500 ppm
I20251026 08:52:30.448689 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:30.449606 413 webserver.cc:492] Webserver started at http://127.0.103.65:39913/ using document root <none> and password file <none>
I20251026 08:52:30.449777 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:30.449828 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:30.449941 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:30.450474 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "9a87b141fa074dfb82294c09a478a650"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.452569 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:30.453819 3378 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.454133 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:30.454216 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "9a87b141fa074dfb82294c09a478a650"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.454298 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:30.467479 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:30.467829 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:30.468221 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:30.468904 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:30.468950 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.469005 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:30.469050 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.479542 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:46019
I20251026 08:52:30.479571 3440 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:46019 every 8 connection(s)
I20251026 08:52:30.480767 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:30.483086 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:30.483327 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:30.484308 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:30.485329 413 server_base.cc:1047] running on GCE node
I20251026 08:52:30.485582 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:30.485651 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:30.485707 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468750485704 us; error 0 us; skew 500 ppm
I20251026 08:52:30.485850 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:30.486987 413 webserver.cc:492] Webserver started at http://127.0.103.66:34249/ using document root <none> and password file <none>
I20251026 08:52:30.487156 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:30.487207 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:30.487215 3441 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36245
I20251026 08:52:30.487314 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:30.487330 3441 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:30.487546 3441 heartbeater.cc:507] Master 127.0.103.126:36245 requested a full tablet report, sending...
I20251026 08:52:30.487814 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "f48002a9efaf4cf89cfecc3b43a6ceca"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.488292 3314 ts_manager.cc:194] Registered new tserver with Master: 9a87b141fa074dfb82294c09a478a650 (127.0.103.65:46019)
I20251026 08:52:30.489189 3314 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53234
I20251026 08:52:30.489895 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:52:30.491230 3453 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.491492 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:30.491569 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "f48002a9efaf4cf89cfecc3b43a6ceca"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.491680 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:30.516904 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:30.517225 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:30.517649 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:30.518308 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:30.518353 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.518389 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:30.518415 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.528862 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:33877
I20251026 08:52:30.528887 3515 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:33877 every 8 connection(s)
I20251026 08:52:30.530148 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:30.532555 3519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:30.532555 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:30.533250 413 server_base.cc:1047] running on GCE node
W20251026 08:52:30.533289 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:30.533649 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:30.533742 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:30.533799 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468750533766 us; error 0 us; skew 500 ppm
I20251026 08:52:30.533959 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:30.535118 413 webserver.cc:492] Webserver started at http://127.0.103.67:45903/ using document root <none> and password file <none>
I20251026 08:52:30.535360 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:30.535471 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:30.535612 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:30.535979 3516 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36245
I20251026 08:52:30.536127 3516 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:30.536268 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "124c5622f7ae4012a7c3fabc875a494d"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.536399 3516 heartbeater.cc:507] Master 127.0.103.126:36245 requested a full tablet report, sending...
I20251026 08:52:30.537128 3314 ts_manager.cc:194] Registered new tserver with Master: f48002a9efaf4cf89cfecc3b43a6ceca (127.0.103.66:33877)
I20251026 08:52:30.537731 3314 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53236
I20251026 08:52:30.538412 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:52:30.539783 3527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.540086 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:30.540179 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "124c5622f7ae4012a7c3fabc875a494d"
format_stamp: "Formatted at 2025-10-26 08:52:30 on dist-test-slave-rd83"
I20251026 08:52:30.540275 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:30.547863 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:30.548189 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:30.548558 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:30.549171 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:30.549230 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.549307 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:30.549357 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:30.558495 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35581
I20251026 08:52:30.558524 3589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35581 every 8 connection(s)
I20251026 08:52:30.564043 3590 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36245
I20251026 08:52:30.564178 3590 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:30.564456 3590 heartbeater.cc:507] Master 127.0.103.126:36245 requested a full tablet report, sending...
I20251026 08:52:30.565219 3314 ts_manager.cc:194] Registered new tserver with Master: 124c5622f7ae4012a7c3fabc875a494d (127.0.103.67:35581)
I20251026 08:52:30.565855 3314 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53242
I20251026 08:52:30.566138 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006785599s
I20251026 08:52:31.490407 3441 heartbeater.cc:499] Master 127.0.103.126:36245 was elected leader, sending a full tablet report...
I20251026 08:52:31.538816 3516 heartbeater.cc:499] Master 127.0.103.126:36245 was elected leader, sending a full tablet report...
I20251026 08:52:31.567003 3590 heartbeater.cc:499] Master 127.0.103.126:36245 was elected leader, sending a full tablet report...
I20251026 08:52:31.603018 413 test_util.cc:276] Using random seed: -415701691
I20251026 08:52:31.614337 3314 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53262:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:52:31.614871 3314 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:52:31.631314 3481 tablet_service.cc:1505] Processing CreateTablet for tablet 7d6ec9e3560a4d339b76ff6394def243 (DEFAULT_TABLE table=test-workload [id=18792ad57b1a465e8eb7cb50b8ea0560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:52:31.631704 3481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d6ec9e3560a4d339b76ff6394def243. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:31.632108 3555 tablet_service.cc:1505] Processing CreateTablet for tablet 7d6ec9e3560a4d339b76ff6394def243 (DEFAULT_TABLE table=test-workload [id=18792ad57b1a465e8eb7cb50b8ea0560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:52:31.632398 3555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d6ec9e3560a4d339b76ff6394def243. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:31.634683 3406 tablet_service.cc:1505] Processing CreateTablet for tablet 7d6ec9e3560a4d339b76ff6394def243 (DEFAULT_TABLE table=test-workload [id=18792ad57b1a465e8eb7cb50b8ea0560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:52:31.635040 3406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d6ec9e3560a4d339b76ff6394def243. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:31.638918 3611 tablet_bootstrap.cc:492] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d: Bootstrap starting.
I20251026 08:52:31.639703 3613 tablet_bootstrap.cc:492] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650: Bootstrap starting.
I20251026 08:52:31.640002 3612 tablet_bootstrap.cc:492] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca: Bootstrap starting.
I20251026 08:52:31.640820 3611 tablet_bootstrap.cc:654] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:31.641402 3613 tablet_bootstrap.cc:654] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:31.641471 3612 tablet_bootstrap.cc:654] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:31.648532 3611 tablet_bootstrap.cc:492] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d: No bootstrap required, opened a new log
I20251026 08:52:31.648680 3613 tablet_bootstrap.cc:492] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650: No bootstrap required, opened a new log
I20251026 08:52:31.648685 3611 ts_tablet_manager.cc:1403] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20251026 08:52:31.648801 3612 tablet_bootstrap.cc:492] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca: No bootstrap required, opened a new log
I20251026 08:52:31.648836 3613 ts_tablet_manager.cc:1403] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20251026 08:52:31.648947 3612 ts_tablet_manager.cc:1403] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251026 08:52:31.649566 3612 raft_consensus.cc:359] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.649520 3611 raft_consensus.cc:359] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.649519 3613 raft_consensus.cc:359] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.649741 3612 raft_consensus.cc:385] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:31.649744 3613 raft_consensus.cc:385] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:31.649750 3611 raft_consensus.cc:385] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:31.649829 3612 raft_consensus.cc:740] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48002a9efaf4cf89cfecc3b43a6ceca, State: Initialized, Role: FOLLOWER
I20251026 08:52:31.649932 3611 raft_consensus.cc:740] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124c5622f7ae4012a7c3fabc875a494d, State: Initialized, Role: FOLLOWER
I20251026 08:52:31.649878 3613 raft_consensus.cc:740] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a87b141fa074dfb82294c09a478a650, State: Initialized, Role: FOLLOWER
I20251026 08:52:31.650141 3612 consensus_queue.cc:260] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.650180 3611 consensus_queue.cc:260] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.650252 3613 consensus_queue.cc:260] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.652611 3613 ts_tablet_manager.cc:1434] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20251026 08:52:31.653437 3612 ts_tablet_manager.cc:1434] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20251026 08:52:31.653714 3611 ts_tablet_manager.cc:1434] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:52:31.867826 3618 raft_consensus.cc:493] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:31.868008 3618 raft_consensus.cc:515] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.868760 3618 leader_election.cc:290] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f48002a9efaf4cf89cfecc3b43a6ceca (127.0.103.66:33877), 9a87b141fa074dfb82294c09a478a650 (127.0.103.65:46019)
I20251026 08:52:31.875090 3491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d6ec9e3560a4d339b76ff6394def243" candidate_uuid: "124c5622f7ae4012a7c3fabc875a494d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" is_pre_election: true
I20251026 08:52:31.875385 3491 raft_consensus.cc:2468] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124c5622f7ae4012a7c3fabc875a494d in term 0.
I20251026 08:52:31.875468 3416 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d6ec9e3560a4d339b76ff6394def243" candidate_uuid: "124c5622f7ae4012a7c3fabc875a494d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a87b141fa074dfb82294c09a478a650" is_pre_election: true
I20251026 08:52:31.875691 3416 raft_consensus.cc:2468] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 124c5622f7ae4012a7c3fabc875a494d in term 0.
I20251026 08:52:31.875842 3531 leader_election.cc:304] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124c5622f7ae4012a7c3fabc875a494d, f48002a9efaf4cf89cfecc3b43a6ceca; no voters:
I20251026 08:52:31.876101 3618 raft_consensus.cc:2804] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:31.876183 3618 raft_consensus.cc:493] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:31.876262 3618 raft_consensus.cc:3060] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:31.878014 3618 raft_consensus.cc:515] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.878500 3618 leader_election.cc:290] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [CANDIDATE]: Term 1 election: Requested vote from peers f48002a9efaf4cf89cfecc3b43a6ceca (127.0.103.66:33877), 9a87b141fa074dfb82294c09a478a650 (127.0.103.65:46019)
I20251026 08:52:31.878926 3416 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d6ec9e3560a4d339b76ff6394def243" candidate_uuid: "124c5622f7ae4012a7c3fabc875a494d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a87b141fa074dfb82294c09a478a650"
I20251026 08:52:31.878926 3491 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7d6ec9e3560a4d339b76ff6394def243" candidate_uuid: "124c5622f7ae4012a7c3fabc875a494d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca"
I20251026 08:52:31.879122 3491 raft_consensus.cc:3060] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:31.879118 3416 raft_consensus.cc:3060] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:31.881745 3491 raft_consensus.cc:2468] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124c5622f7ae4012a7c3fabc875a494d in term 1.
I20251026 08:52:31.881745 3416 raft_consensus.cc:2468] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124c5622f7ae4012a7c3fabc875a494d in term 1.
I20251026 08:52:31.882140 3531 leader_election.cc:304] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124c5622f7ae4012a7c3fabc875a494d, f48002a9efaf4cf89cfecc3b43a6ceca; no voters:
I20251026 08:52:31.882341 3618 raft_consensus.cc:2804] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:31.882714 3618 raft_consensus.cc:697] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 1 LEADER]: Becoming Leader. State: Replica: 124c5622f7ae4012a7c3fabc875a494d, State: Running, Role: LEADER
I20251026 08:52:31.882941 3618 consensus_queue.cc:237] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } }
I20251026 08:52:31.885039 3313 catalog_manager.cc:5649] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d reported cstate change: term changed from 0 to 1, leader changed from <none> to 124c5622f7ae4012a7c3fabc875a494d (127.0.103.67). New cstate: current_term: 1 leader_uuid: "124c5622f7ae4012a7c3fabc875a494d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "124c5622f7ae4012a7c3fabc875a494d" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:32.412515 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:32.412729 3367 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:32.413118 3367 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:32.413216 3367 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:32.458261 3624 consensus_queue.cc:1048] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a87b141fa074dfb82294c09a478a650" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 46019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:32.463039 3627 consensus_queue.cc:1048] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48002a9efaf4cf89cfecc3b43a6ceca" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:33.415052 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:34.413543 3367 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:34.413946 3367 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:34.414023 3367 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:34.417086 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:35.419385 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:36.414494 3367 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:36.414876 3367 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:36.414955 3367 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:36.421437 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:37.423573 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:38.415290 3367 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:38.415665 3367 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:38.415753 3367 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:38.425586 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:39.427698 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:40.416138 3367 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:40.416553 3367 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:40.416626 3367 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:40.429777 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:41.431983 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:42.416962 3367 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:42.417392 3367 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:42.417464 3367 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:42.434016 3362 placement_policy_util.cc:407] tablet 7d6ec9e3560a4d339b76ff6394def243: detected majority of replicas (2 of 3) at location L0
I20251026 08:52:42.968624 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:52:42.983984 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:42.984289 413 tablet_replica.cc:333] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650: stopping tablet replica
I20251026 08:52:42.984498 413 raft_consensus.cc:2243] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:42.984680 413 raft_consensus.cc:2272] T 7d6ec9e3560a4d339b76ff6394def243 P 9a87b141fa074dfb82294c09a478a650 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:42.998651 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:52:43.001166 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:52:43.009042 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:43.009268 413 tablet_replica.cc:333] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca: stopping tablet replica
I20251026 08:52:43.009431 413 raft_consensus.cc:2243] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:43.009582 413 raft_consensus.cc:2272] T 7d6ec9e3560a4d339b76ff6394def243 P f48002a9efaf4cf89cfecc3b43a6ceca [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:43.022693 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:52:43.025014 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:52:43.032924 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:52:43.033212 413 tablet_replica.cc:333] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d: stopping tablet replica
I20251026 08:52:43.033386 413 raft_consensus.cc:2243] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:52:43.033663 413 raft_consensus.cc:2272] T 7d6ec9e3560a4d339b76ff6394def243 P 124c5622f7ae4012a7c3fabc875a494d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:43.036507 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:52:43.038901 413 master.cc:561] Master@127.0.103.126:36245 shutting down...
I20251026 08:52:43.045544 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:52:43.045740 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:43.045836 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 518eb071f47b44429fc2c70bdb8c14f3: stopping tablet replica
I20251026 08:52:43.058693 413 master.cc:583] Master@127.0.103.126:36245 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (12708 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
I20251026 08:52:43.068351 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:39299
I20251026 08:52:43.068758 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:43.071022 3634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.071151 3635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.071221 3637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:43.071478 413 server_base.cc:1047] running on GCE node
I20251026 08:52:43.072347 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:43.072408 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:43.072448 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468763072437 us; error 0 us; skew 500 ppm
I20251026 08:52:43.072580 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:43.073633 413 webserver.cc:492] Webserver started at http://127.0.103.126:38703/ using document root <none> and password file <none>
I20251026 08:52:43.073808 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:43.073861 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:43.073949 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:43.074501 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "69d6bf04204a4fafb6d6c224711cc3c8"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.076508 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20251026 08:52:43.077804 3642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.078094 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:43.078191 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "69d6bf04204a4fafb6d6c224711cc3c8"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.078303 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:43.088632 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:43.089102 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:43.099614 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:39299
I20251026 08:52:43.099632 3693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:39299 every 8 connection(s)
I20251026 08:52:43.100749 3694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:43.103962 3694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8: Bootstrap starting.
I20251026 08:52:43.104933 3694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:43.106182 3694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8: No bootstrap required, opened a new log
I20251026 08:52:43.106694 3694 raft_consensus.cc:359] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" member_type: VOTER }
I20251026 08:52:43.106820 3694 raft_consensus.cc:385] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:43.106874 3694 raft_consensus.cc:740] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d6bf04204a4fafb6d6c224711cc3c8, State: Initialized, Role: FOLLOWER
I20251026 08:52:43.107028 3694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" member_type: VOTER }
I20251026 08:52:43.107137 3694 raft_consensus.cc:399] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:52:43.107188 3694 raft_consensus.cc:493] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:52:43.107265 3694 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:43.108651 3694 raft_consensus.cc:515] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" member_type: VOTER }
I20251026 08:52:43.108805 3694 leader_election.cc:304] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d6bf04204a4fafb6d6c224711cc3c8; no voters:
I20251026 08:52:43.109117 3694 leader_election.cc:290] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:52:43.109215 3697 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:43.109659 3697 raft_consensus.cc:697] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 1 LEADER]: Becoming Leader. State: Replica: 69d6bf04204a4fafb6d6c224711cc3c8, State: Running, Role: LEADER
I20251026 08:52:43.109925 3694 sys_catalog.cc:565] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:52:43.109890 3697 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" member_type: VOTER }
I20251026 08:52:43.110648 3699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d6bf04204a4fafb6d6c224711cc3c8. Latest consensus state: current_term: 1 leader_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" member_type: VOTER } }
I20251026 08:52:43.111035 3699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:43.111038 3698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d6bf04204a4fafb6d6c224711cc3c8" member_type: VOTER } }
I20251026 08:52:43.111469 3698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [sys.catalog]: This master's current role is: LEADER
I20251026 08:52:43.112643 3703 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:52:43.114641 3703 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:52:43.116237 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:52:43.117733 3703 catalog_manager.cc:1357] Generated new cluster ID: de1b651ae00640d1935b36fe9dca6e8d
I20251026 08:52:43.117813 3703 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:52:43.123806 3703 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:52:43.124501 3703 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:52:43.133417 3703 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8: Generated new TSK 0
I20251026 08:52:43.133602 3703 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:52:43.148661 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:43.151155 3715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.151343 3716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.151700 3718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:43.151868 413 server_base.cc:1047] running on GCE node
I20251026 08:52:43.152132 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:43.152195 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:43.152256 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468763152235 us; error 0 us; skew 500 ppm
I20251026 08:52:43.152395 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:43.153396 413 webserver.cc:492] Webserver started at http://127.0.103.65:41227/ using document root <none> and password file <none>
I20251026 08:52:43.153601 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:43.153672 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:43.153807 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:43.154402 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "e7be295d3ba247a1a4475fa020d1ade1"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.156625 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20251026 08:52:43.158056 3723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.158406 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:52:43.158514 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "e7be295d3ba247a1a4475fa020d1ade1"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.158623 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:43.187920 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:43.188351 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:43.188822 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:43.189594 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:43.189646 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.189713 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:43.189755 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.199702 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:39515
I20251026 08:52:43.199726 3785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:39515 every 8 connection(s)
I20251026 08:52:43.201359 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:43.203668 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.204255 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.205049 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:43.205895 413 server_base.cc:1047] running on GCE node
I20251026 08:52:43.206128 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:43.206183 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:43.206209 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468763206203 us; error 0 us; skew 500 ppm
I20251026 08:52:43.206347 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:43.207599 413 webserver.cc:492] Webserver started at http://127.0.103.66:35627/ using document root <none> and password file <none>
I20251026 08:52:43.207803 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:43.207866 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:43.207962 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:43.208519 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.210738 3786 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39299
I20251026 08:52:43.210878 3786 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:43.211119 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:43.211184 3786 heartbeater.cc:507] Master 127.0.103.126:39299 requested a full tablet report, sending...
I20251026 08:52:43.212051 3659 ts_manager.cc:194] Registered new tserver with Master: e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:43.212705 3798 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.212970 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20251026 08:52:43.213047 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.213129 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:43.213372 3659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50746
I20251026 08:52:43.233258 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:43.233645 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:43.234020 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:43.234612 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:43.234660 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.234716 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:43.234750 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.244498 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:37913
I20251026 08:52:43.244521 3860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:37913 every 8 connection(s)
I20251026 08:52:43.245741 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:43.247973 3864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.247990 3865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:43.248746 3867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:43.249899 413 server_base.cc:1047] running on GCE node
I20251026 08:52:43.250180 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:43.250262 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:43.250309 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468763250305 us; error 0 us; skew 500 ppm
I20251026 08:52:43.250458 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:43.251587 413 webserver.cc:492] Webserver started at http://127.0.103.67:45033/ using document root <none> and password file <none>
I20251026 08:52:43.251752 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:43.251787 3861 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39299
I20251026 08:52:43.251816 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:43.251897 3861 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:43.251938 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:43.252187 3861 heartbeater.cc:507] Master 127.0.103.126:39299 requested a full tablet report, sending...
I20251026 08:52:43.252575 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.252956 3659 ts_manager.cc:194] Registered new tserver with Master: 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:43.253813 3659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50758
I20251026 08:52:43.255076 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:43.256243 3872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.256549 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:43.256644 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
format_stamp: "Formatted at 2025-10-26 08:52:43 on dist-test-slave-rd83"
I20251026 08:52:43.256728 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:43.270112 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:43.270439 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:43.270823 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:43.271440 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:43.271487 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.271543 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:43.271574 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:43.280951 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:33171
I20251026 08:52:43.280978 3934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:33171 every 8 connection(s)
I20251026 08:52:43.285992 3935 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39299
I20251026 08:52:43.286132 3935 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:43.286396 3935 heartbeater.cc:507] Master 127.0.103.126:39299 requested a full tablet report, sending...
I20251026 08:52:43.287142 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005543945s
I20251026 08:52:43.287123 3659 ts_manager.cc:194] Registered new tserver with Master: 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:43.287866 3659 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50762
I20251026 08:52:44.214867 3786 heartbeater.cc:499] Master 127.0.103.126:39299 was elected leader, sending a full tablet report...
I20251026 08:52:44.255031 3861 heartbeater.cc:499] Master 127.0.103.126:39299 was elected leader, sending a full tablet report...
I20251026 08:52:44.288968 3935 heartbeater.cc:499] Master 127.0.103.126:39299 was elected leader, sending a full tablet report...
I20251026 08:52:44.314112 413 test_util.cc:276] Using random seed: -402990652
I20251026 08:52:44.328444 3659 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50774:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu""\004\001\000\252\252\252\n\004\001\000TUU\025\004\001\000\376\377\377\037\004\001\000\250\252\252*\004\001\000RUU5\004\001\000\374\377\377?\004\001\000\246\252\252J\004\001\000PUUU\004\001\000\372\377\377_\004\001\000\244\252\252j\004\001\000NUUu"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:52:44.329245 3659 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:52:44.350641 3900 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3135b8b6fe4dff81329c307264ea9f (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251026 08:52:44.351037 3900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3135b8b6fe4dff81329c307264ea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.354447 3899 tablet_service.cc:1505] Processing CreateTablet for tablet 85fe4797e56f4170876f76414e23628f (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251026 08:52:44.354629 3826 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3135b8b6fe4dff81329c307264ea9f (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251026 08:52:44.354676 3891 tablet_service.cc:1505] Processing CreateTablet for tablet c1e736aea2b54b378ffbe821cb9e42c4 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251026 08:52:44.354835 3899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe4797e56f4170876f76414e23628f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.355016 3891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e736aea2b54b378ffbe821cb9e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.355022 3826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3135b8b6fe4dff81329c307264ea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.358422 3825 tablet_service.cc:1505] Processing CreateTablet for tablet 85fe4797e56f4170876f76414e23628f (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251026 08:52:44.358770 3825 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe4797e56f4170876f76414e23628f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.358718 3893 tablet_service.cc:1505] Processing CreateTablet for tablet 5f3c583b37d74efe9630e3723a2c313e (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251026 08:52:44.359115 3893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3c583b37d74efe9630e3723a2c313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.358422 3892 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee117517b8b4ae18fa61970868914a4 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251026 08:52:44.362459 3892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee117517b8b4ae18fa61970868914a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.362743 3895 tablet_service.cc:1505] Processing CreateTablet for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251026 08:52:44.363027 3895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.364674 3899 tablet_service.cc:1505] Processing CreateTablet for tablet 6dd57692a7704c9690a892319e4bf446 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251026 08:52:44.365074 3899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd57692a7704c9690a892319e4bf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.362381 3894 tablet_service.cc:1505] Processing CreateTablet for tablet b1edddc92bb0463b9ed275da3c7154ef (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251026 08:52:44.366333 3894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1edddc92bb0463b9ed275da3c7154ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.366380 3896 tablet_service.cc:1505] Processing CreateTablet for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251026 08:52:44.366531 3955 tablet_bootstrap.cc:492] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.366744 3896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f200f20b1741b4a2c6bc379b2ae8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.367995 3955 tablet_bootstrap.cc:654] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.370385 3897 tablet_service.cc:1505] Processing CreateTablet for tablet d0a49b69e0a94ae9827e91c43006c2d5 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251026 08:52:44.370479 3824 tablet_service.cc:1505] Processing CreateTablet for tablet 0409b48759d34a6fa9c86d88b809e093 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251026 08:52:44.370716 3897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a49b69e0a94ae9827e91c43006c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.370813 3824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0409b48759d34a6fa9c86d88b809e093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.371846 3898 tablet_service.cc:1505] Processing CreateTablet for tablet 0409b48759d34a6fa9c86d88b809e093 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251026 08:52:44.372380 3898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0409b48759d34a6fa9c86d88b809e093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.374538 3823 tablet_service.cc:1505] Processing CreateTablet for tablet d0a49b69e0a94ae9827e91c43006c2d5 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251026 08:52:44.374872 3823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a49b69e0a94ae9827e91c43006c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.380468 3822 tablet_service.cc:1505] Processing CreateTablet for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251026 08:52:44.380662 3957 tablet_bootstrap.cc:492] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.380905 3822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f200f20b1741b4a2c6bc379b2ae8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.387228 3823 tablet_service.cc:1505] Processing CreateTablet for tablet 5f3c583b37d74efe9630e3723a2c313e (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251026 08:52:44.387339 3821 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee117517b8b4ae18fa61970868914a4 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251026 08:52:44.387593 3823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3c583b37d74efe9630e3723a2c313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.387732 3821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee117517b8b4ae18fa61970868914a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.390689 3820 tablet_service.cc:1505] Processing CreateTablet for tablet c1e736aea2b54b378ffbe821cb9e42c4 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251026 08:52:44.391247 3819 tablet_service.cc:1505] Processing CreateTablet for tablet 6dd57692a7704c9690a892319e4bf446 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251026 08:52:44.391579 3819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd57692a7704c9690a892319e4bf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.392057 3820 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e736aea2b54b378ffbe821cb9e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.392563 3818 tablet_service.cc:1505] Processing CreateTablet for tablet 8c430afe1db8423b97c3c436cb912760 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251026 08:52:44.392866 3818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c430afe1db8423b97c3c436cb912760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.395150 3824 tablet_service.cc:1505] Processing CreateTablet for tablet b1edddc92bb0463b9ed275da3c7154ef (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251026 08:52:44.395440 3824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1edddc92bb0463b9ed275da3c7154ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.395396 3898 tablet_service.cc:1505] Processing CreateTablet for tablet 8c430afe1db8423b97c3c436cb912760 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251026 08:52:44.395644 3898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c430afe1db8423b97c3c436cb912760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.398378 3822 tablet_service.cc:1505] Processing CreateTablet for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251026 08:52:44.398667 3822 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.401340 3957 tablet_bootstrap.cc:654] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.404534 3957 tablet_bootstrap.cc:492] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.404649 3957 ts_tablet_manager.cc:1403] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251026 08:52:44.405284 3957 raft_consensus.cc:359] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.405436 3957 raft_consensus.cc:385] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.405489 3957 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.405673 3957 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.406697 3751 tablet_service.cc:1505] Processing CreateTablet for tablet 3d3135b8b6fe4dff81329c307264ea9f (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION VALUES < 178956970
I20251026 08:52:44.407054 3955 tablet_bootstrap.cc:492] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.407173 3955 ts_tablet_manager.cc:1403] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.024s
I20251026 08:52:44.407181 3751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3135b8b6fe4dff81329c307264ea9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.407853 3955 raft_consensus.cc:359] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.407984 3955 raft_consensus.cc:385] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.408022 3955 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.408269 3955 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.408905 3957 ts_tablet_manager.cc:1434] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.004s user 0.002s sys 0.002s
I20251026 08:52:44.409672 3957 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.409770 3750 tablet_service.cc:1505] Processing CreateTablet for tablet 85fe4797e56f4170876f76414e23628f (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 178956970 <= VALUES < 357913940
I20251026 08:52:44.410145 3750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe4797e56f4170876f76414e23628f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.411676 3957 tablet_bootstrap.cc:654] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.416599 3749 tablet_service.cc:1505] Processing CreateTablet for tablet 0409b48759d34a6fa9c86d88b809e093 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 357913940 <= VALUES < 536870910
I20251026 08:52:44.417021 3749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0409b48759d34a6fa9c86d88b809e093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.418691 3957 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.418817 3957 ts_tablet_manager.cc:1403] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20251026 08:52:44.419100 3751 tablet_service.cc:1505] Processing CreateTablet for tablet d0a49b69e0a94ae9827e91c43006c2d5 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 715827880
I20251026 08:52:44.419478 3957 raft_consensus.cc:359] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.419632 3957 raft_consensus.cc:385] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.419682 3957 raft_consensus.cc:740] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.419567 3751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a49b69e0a94ae9827e91c43006c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.420136 3748 tablet_service.cc:1505] Processing CreateTablet for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 715827880 <= VALUES < 894784850
I20251026 08:52:44.420598 3748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f200f20b1741b4a2c6bc379b2ae8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.422853 3957 consensus_queue.cc:260] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.425101 3957 ts_tablet_manager.cc:1434] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 08:52:44.425848 3957 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.427211 3957 tablet_bootstrap.cc:654] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.427348 3751 tablet_service.cc:1505] Processing CreateTablet for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 894784850 <= VALUES < 1073741820
I20251026 08:52:44.427654 3751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.428462 3749 tablet_service.cc:1505] Processing CreateTablet for tablet b1edddc92bb0463b9ed275da3c7154ef (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1252698790
I20251026 08:52:44.428745 3749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1edddc92bb0463b9ed275da3c7154ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.428699 3748 tablet_service.cc:1505] Processing CreateTablet for tablet 5f3c583b37d74efe9630e3723a2c313e (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1252698790 <= VALUES < 1431655760
I20251026 08:52:44.429056 3748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3c583b37d74efe9630e3723a2c313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.434218 3962 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.434665 3747 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee117517b8b4ae18fa61970868914a4 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1431655760 <= VALUES < 1610612730
I20251026 08:52:44.435328 3747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee117517b8b4ae18fa61970868914a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.435894 3962 tablet_bootstrap.cc:654] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.437503 3955 ts_tablet_manager.cc:1434] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.030s user 0.008s sys 0.021s
I20251026 08:52:44.437788 3955 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.438148 3746 tablet_service.cc:1505] Processing CreateTablet for tablet c1e736aea2b54b378ffbe821cb9e42c4 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1789569700
I20251026 08:52:44.438575 3746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e736aea2b54b378ffbe821cb9e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.438980 3959 raft_consensus.cc:493] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.439141 3959 raft_consensus.cc:515] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.443907 3745 tablet_service.cc:1505] Processing CreateTablet for tablet 6dd57692a7704c9690a892319e4bf446 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1789569700 <= VALUES < 1968526670
I20251026 08:52:44.444345 3745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd57692a7704c9690a892319e4bf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.444444 3959 leader_election.cc:290] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:44.447355 3744 tablet_service.cc:1505] Processing CreateTablet for tablet 8c430afe1db8423b97c3c436cb912760 (DEFAULT_TABLE table=test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]), partition=RANGE (key) PARTITION 1968526670 <= VALUES
I20251026 08:52:44.447853 3744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c430afe1db8423b97c3c436cb912760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:44.453683 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.453917 3910 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.454491 3801 leader_election.cc:304] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:44.454839 3959 raft_consensus.cc:2804] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.454952 3959 raft_consensus.cc:493] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.455047 3959 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.457990 3959 raft_consensus.cc:515] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.458683 3955 tablet_bootstrap.cc:654] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.459086 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.459293 3910 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.461684 3910 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.461918 3955 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.462108 3955 ts_tablet_manager.cc:1403] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20251026 08:52:44.462285 3801 leader_election.cc:304] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:44.462847 3955 raft_consensus.cc:359] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.463099 3955 raft_consensus.cc:385] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.463213 3955 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.463428 3955 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.463896 3955 ts_tablet_manager.cc:1434] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:52:44.464205 3955 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.465962 3955 tablet_bootstrap.cc:654] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.469437 3959 leader_election.cc:290] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:44.470117 3959 raft_consensus.cc:2804] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.470582 3959 raft_consensus.cc:697] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:44.471081 3959 consensus_queue.cc:237] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.483430 3960 raft_consensus.cc:493] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.483628 3960 raft_consensus.cc:515] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.484800 3960 leader_election.cc:290] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.485682 3957 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.485867 3957 ts_tablet_manager.cc:1403] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.000s
I20251026 08:52:44.489569 3957 raft_consensus.cc:359] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.489810 3957 raft_consensus.cc:385] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.489928 3957 raft_consensus.cc:740] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.490195 3957 consensus_queue.cc:260] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.490831 3957 ts_tablet_manager.cc:1434] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251026 08:52:44.491142 3957 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.493614 3761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.495220 3962 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.495503 3962 ts_tablet_manager.cc:1403] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.030s
I20251026 08:52:44.495800 3955 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.495944 3955 ts_tablet_manager.cc:1403] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.021s
I20251026 08:52:44.494500 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.496381 3962 raft_consensus.cc:359] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.496575 3962 raft_consensus.cc:385] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.496331 3659 catalog_manager.cc:5649] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:44.496663 3962 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.496721 3955 raft_consensus.cc:359] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.496874 3955 raft_consensus.cc:385] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.496974 3955 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.496877 3962 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.498051 3957 tablet_bootstrap.cc:654] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.498679 3955 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
W20251026 08:52:44.499752 3802 leader_election.cc:343] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.500627 3955 ts_tablet_manager.cc:1434] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20251026 08:52:44.500981 3955 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
W20251026 08:52:44.502727 3802 leader_election.cc:343] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.503717 3955 tablet_bootstrap.cc:654] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.505393 3962 ts_tablet_manager.cc:1434] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20251026 08:52:44.505694 3962 tablet_bootstrap.cc:492] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.507382 3962 tablet_bootstrap.cc:654] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.511580 3955 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.511735 3955 ts_tablet_manager.cc:1403] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20251026 08:52:44.512429 3955 raft_consensus.cc:359] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.512609 3955 raft_consensus.cc:385] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.512728 3955 raft_consensus.cc:740] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.513005 3955 consensus_queue.cc:260] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.513465 3955 ts_tablet_manager.cc:1434] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:52:44.513736 3955 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.515938 3955 tablet_bootstrap.cc:654] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.516208 3962 tablet_bootstrap.cc:492] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.516393 3962 ts_tablet_manager.cc:1403] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20251026 08:52:44.516960 3962 raft_consensus.cc:359] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.517132 3962 raft_consensus.cc:385] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.517220 3962 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.517436 3962 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:44.517673 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.517863 3962 ts_tablet_manager.cc:1434] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:52:44.518127 3962 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
W20251026 08:52:44.518991 3875 leader_election.cc:343] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.519763 3962 tablet_bootstrap.cc:654] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.520012 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.520248 3760 raft_consensus.cc:2468] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 0.
I20251026 08:52:44.520718 3876 leader_election.cc:304] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters: 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:52:44.521194 3960 raft_consensus.cc:2804] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.521306 3960 raft_consensus.cc:493] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.521404 3960 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.524868 3960 raft_consensus.cc:515] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.525940 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.526173 3760 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.529744 3760 raft_consensus.cc:2468] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 1.
I20251026 08:52:44.530221 3876 leader_election.cc:304] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.530232 3960 leader_election.cc:290] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.531905 3960 raft_consensus.cc:2804] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.532038 3960 raft_consensus.cc:697] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:44.532338 3960 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.533419 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
W20251026 08:52:44.533985 3875 leader_election.cc:343] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.535571 3962 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.535703 3962 ts_tablet_manager.cc:1403] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.011s
I20251026 08:52:44.536041 3955 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.536130 3957 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.536194 3955 ts_tablet_manager.cc:1403] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.006s
I20251026 08:52:44.536274 3957 ts_tablet_manager.cc:1403] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.045s user 0.008s sys 0.001s
I20251026 08:52:44.536381 3962 raft_consensus.cc:359] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.536545 3962 raft_consensus.cc:385] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.536613 3962 raft_consensus.cc:740] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.536799 3962 consensus_queue.cc:260] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.536942 3957 raft_consensus.cc:359] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.537065 3955 raft_consensus.cc:359] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.537256 3962 ts_tablet_manager.cc:1434] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251026 08:52:44.537302 3955 raft_consensus.cc:385] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.537410 3955 raft_consensus.cc:740] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.537528 3962 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.537613 3955 consensus_queue.cc:260] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.537081 3957 raft_consensus.cc:385] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.538033 3957 raft_consensus.cc:740] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.538292 3957 consensus_queue.cc:260] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.538884 3957 ts_tablet_manager.cc:1434] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:52:44.539147 3962 tablet_bootstrap.cc:654] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.539249 3957 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.541180 3957 tablet_bootstrap.cc:654] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.547629 3955 ts_tablet_manager.cc:1434] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20251026 08:52:44.547922 3955 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.548625 3957 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.549096 3962 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.549100 3957 ts_tablet_manager.cc:1403] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20251026 08:52:44.549248 3962 ts_tablet_manager.cc:1403] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.003s
I20251026 08:52:44.549209 3657 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:44.549896 3955 tablet_bootstrap.cc:654] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.549917 3962 raft_consensus.cc:359] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.550117 3962 raft_consensus.cc:385] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.550030 3957 raft_consensus.cc:359] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.550212 3962 raft_consensus.cc:740] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.550371 3957 raft_consensus.cc:385] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.550503 3957 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.550469 3962 consensus_queue.cc:260] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.550779 3957 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.550993 3962 ts_tablet_manager.cc:1434] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:44.551308 3962 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.551287 3957 ts_tablet_manager.cc:1434] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:52:44.552386 3957 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.553344 3962 tablet_bootstrap.cc:654] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.554360 3957 tablet_bootstrap.cc:654] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.555213 3955 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.555388 3955 ts_tablet_manager.cc:1403] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20251026 08:52:44.556043 3955 raft_consensus.cc:359] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.556316 3955 raft_consensus.cc:385] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.556450 3955 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.556736 3955 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.557360 3955 ts_tablet_manager.cc:1434] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:52:44.557739 3973 raft_consensus.cc:493] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.557893 3973 raft_consensus.cc:515] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.558967 3955 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.559185 3973 leader_election.cc:290] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.561242 3955 tablet_bootstrap.cc:654] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.562531 3962 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.562672 3962 ts_tablet_manager.cc:1403] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20251026 08:52:44.562973 3957 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.563122 3957 ts_tablet_manager.cc:1403] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20251026 08:52:44.563642 3962 raft_consensus.cc:359] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.563902 3962 raft_consensus.cc:385] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.564054 3962 raft_consensus.cc:740] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.564015 3957 raft_consensus.cc:359] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.564244 3957 raft_consensus.cc:385] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.564368 3962 consensus_queue.cc:260] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.565112 3962 ts_tablet_manager.cc:1434] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:44.565531 3962 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.567967 3962 tablet_bootstrap.cc:654] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.564378 3957 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.569968 3957 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.570650 3957 ts_tablet_manager.cc:1434] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20251026 08:52:44.570991 3957 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.573168 3957 tablet_bootstrap.cc:654] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.579821 3957 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.579977 3957 ts_tablet_manager.cc:1403] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20251026 08:52:44.580327 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.580612 3957 raft_consensus.cc:359] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.580768 3957 raft_consensus.cc:385] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.580854 3957 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.580935 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.581092 3836 raft_consensus.cc:2468] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.581068 3957 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
W20251026 08:52:44.581435 3726 leader_election.cc:343] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.581933 3957 ts_tablet_manager.cc:1434] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:52:44.582226 3957 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.583676 3957 tablet_bootstrap.cc:654] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.581621 3726 leader_election.cc:304] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters: 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:44.585242 3973 raft_consensus.cc:2804] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.585373 3973 raft_consensus.cc:493] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.585492 3973 raft_consensus.cc:3060] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.589341 3973 raft_consensus.cc:515] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.590041 3973 leader_election.cc:290] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.590776 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:44.591003 3836 raft_consensus.cc:3060] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.594954 3959 raft_consensus.cc:493] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.595129 3959 raft_consensus.cc:515] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.595744 3959 leader_election.cc:290] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.596401 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.596638 3910 raft_consensus.cc:2468] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.597227 3801 leader_election.cc:304] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:44.597563 3959 raft_consensus.cc:2804] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.597671 3959 raft_consensus.cc:493] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.597774 3959 raft_consensus.cc:3060] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.601040 3959 raft_consensus.cc:515] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.602300 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.602495 3910 raft_consensus.cc:3060] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.605484 3910 raft_consensus.cc:2468] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.605882 3801 leader_election.cc:304] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:44.606276 3968 raft_consensus.cc:2804] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.608100 3836 raft_consensus.cc:2468] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 1.
I20251026 08:52:44.610200 3959 leader_election.cc:290] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.615703 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
W20251026 08:52:44.616412 3726 leader_election.cc:343] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.616665 3726 leader_election.cc:304] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters: 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:44.617619 3973 raft_consensus.cc:2804] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.622323 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.622522 3760 raft_consensus.cc:2468] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.623275 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.623461 3760 raft_consensus.cc:3060] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.627159 3760 raft_consensus.cc:2468] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.634253 3962 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.634689 3962 ts_tablet_manager.cc:1403] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.069s user 0.027s sys 0.034s
I20251026 08:52:44.636194 3962 raft_consensus.cc:359] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.636466 3962 raft_consensus.cc:385] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.636575 3962 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.636907 3962 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.637847 3973 raft_consensus.cc:697] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:44.638224 3962 ts_tablet_manager.cc:1434] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251026 08:52:44.638269 3973 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.638718 3962 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.641638 3962 tablet_bootstrap.cc:654] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.644030 3968 raft_consensus.cc:697] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:44.644320 3955 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.644493 3955 ts_tablet_manager.cc:1403] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.086s user 0.023s sys 0.059s
I20251026 08:52:44.644395 3968 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.645395 3955 raft_consensus.cc:359] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.645568 3955 raft_consensus.cc:385] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.645642 3955 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.645866 3955 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.646708 3955 ts_tablet_manager.cc:1434] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:44.647156 3955 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.649617 3955 tablet_bootstrap.cc:654] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.651791 3962 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.651878 3957 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.652060 3957 ts_tablet_manager.cc:1403] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.070s user 0.034s sys 0.030s
I20251026 08:52:44.652909 3957 raft_consensus.cc:359] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.653106 3957 raft_consensus.cc:385] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.653192 3957 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.653395 3957 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.653864 3957 ts_tablet_manager.cc:1434] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:52:44.653798 3657 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.652066 3962 ts_tablet_manager.cc:1403] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20251026 08:52:44.654127 3957 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.654834 3962 raft_consensus.cc:359] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.655021 3962 raft_consensus.cc:385] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.655238 3962 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.655571 3962 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.655856 3957 tablet_bootstrap.cc:654] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.656062 3955 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.656165 3962 ts_tablet_manager.cc:1434] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:52:44.656214 3955 ts_tablet_manager.cc:1403] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20251026 08:52:44.656740 3962 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.656735 3658 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.657388 3955 raft_consensus.cc:359] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.657608 3955 raft_consensus.cc:385] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.657718 3955 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.657965 3955 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.658483 3955 ts_tablet_manager.cc:1434] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:44.658663 3962 tablet_bootstrap.cc:654] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.658754 3955 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.658795 3993 raft_consensus.cc:493] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.658990 3993 raft_consensus.cc:515] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.659895 3993 leader_election.cc:290] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.660709 3973 raft_consensus.cc:493] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.660830 3973 raft_consensus.cc:515] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.661059 3955 tablet_bootstrap.cc:654] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.661556 3973 leader_election.cc:290] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.661718 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.661926 3910 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.662322 3909 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.662526 3909 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.662607 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.662832 3836 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.665777 3726 leader_election.cc:304] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.666589 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.666795 3760 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.669260 3801 leader_election.cc:304] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:44.671207 3955 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.671391 3955 ts_tablet_manager.cc:1403] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20251026 08:52:44.672281 3955 raft_consensus.cc:359] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.672498 3955 raft_consensus.cc:385] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.672607 3993 raft_consensus.cc:2804] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.673226 3993 raft_consensus.cc:493] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.673358 3973 raft_consensus.cc:2804] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.673404 3993 raft_consensus.cc:3060] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.673487 3973 raft_consensus.cc:493] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.673584 3973 raft_consensus.cc:3060] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.677130 3973 raft_consensus.cc:515] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.677189 3993 raft_consensus.cc:515] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.678364 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:44.678709 3836 raft_consensus.cc:2393] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7be295d3ba247a1a4475fa020d1ade1 in current term 1: Already voted for candidate 96cd069722b34c1eab8ae9a2f953dd7a in this term.
I20251026 08:52:44.679006 3973 leader_election.cc:290] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.680583 3962 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.680666 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.680752 3962 ts_tablet_manager.cc:1403] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.015s
I20251026 08:52:44.680747 3957 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.680874 3760 raft_consensus.cc:2393] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96cd069722b34c1eab8ae9a2f953dd7a in current term 1: Already voted for candidate e7be295d3ba247a1a4475fa020d1ade1 in this term.
I20251026 08:52:44.680997 3957 ts_tablet_manager.cc:1403] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.006s
I20251026 08:52:44.681205 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.681365 3910 raft_consensus.cc:3060] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.681579 3962 raft_consensus.cc:359] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.681769 3962 raft_consensus.cc:385] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.681725 3957 raft_consensus.cc:359] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.681860 3962 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.681903 3957 raft_consensus.cc:385] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.681993 3957 raft_consensus.cc:740] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.682088 3962 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.682224 3957 consensus_queue.cc:260] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.682621 3962 ts_tablet_manager.cc:1434] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:52:44.682662 3957 ts_tablet_manager.cc:1434] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:52:44.682885 3962 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.682924 3957 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.684212 3910 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 1.
I20251026 08:52:44.684669 3957 tablet_bootstrap.cc:654] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.684664 3726 leader_election.cc:304] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters: 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:52:44.684774 3962 tablet_bootstrap.cc:654] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.685137 3973 raft_consensus.cc:2804] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.685344 3960 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.685298 3973 raft_consensus.cc:697] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:44.685822 3960 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.686079 3973 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.686460 3960 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.686993 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.687160 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
W20251026 08:52:44.687532 3876 leader_election.cc:343] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): Illegal state: must be running to vote when last-logged opid is not known
W20251026 08:52:44.687770 3875 leader_election.cc:343] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:52:44.687913 3875 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae; no voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:44.688431 3960 raft_consensus.cc:2749] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 08:52:44.689013 3993 leader_election.cc:290] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.673221 3955 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.689317 3955 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.689815 3955 ts_tablet_manager.cc:1434] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.018s user 0.001s sys 0.002s
I20251026 08:52:44.690176 3955 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.692553 3955 tablet_bootstrap.cc:654] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.694862 3957 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.694990 3957 ts_tablet_manager.cc:1403] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20251026 08:52:44.695434 3657 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.695611 3957 raft_consensus.cc:359] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.695744 3957 raft_consensus.cc:385] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.695847 3957 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.696051 3957 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.696183 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.696391 3910 raft_consensus.cc:2393] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 96cd069722b34c1eab8ae9a2f953dd7a in current term 1: Already voted for candidate e7be295d3ba247a1a4475fa020d1ade1 in this term.
I20251026 08:52:44.696655 3957 ts_tablet_manager.cc:1434] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:52:44.696740 3801 leader_election.cc:304] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a; no voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:44.696918 3957 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.697088 3962 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.697135 3993 raft_consensus.cc:2749] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251026 08:52:44.697306 3962 ts_tablet_manager.cc:1403] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.009s
I20251026 08:52:44.698733 3962 raft_consensus.cc:359] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.699031 3962 raft_consensus.cc:385] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.699152 3957 tablet_bootstrap.cc:654] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.699147 3962 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.699586 3962 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.700448 3962 ts_tablet_manager.cc:1434] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251026 08:52:44.700799 3962 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.704063 3962 tablet_bootstrap.cc:654] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.706267 3955 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.706511 3955 ts_tablet_manager.cc:1403] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.012s
I20251026 08:52:44.707022 3957 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.707165 3957 ts_tablet_manager.cc:1403] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20251026 08:52:44.707561 3955 raft_consensus.cc:359] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.707706 3957 raft_consensus.cc:359] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.707810 3955 raft_consensus.cc:385] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.707857 3957 raft_consensus.cc:385] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.707916 3955 raft_consensus.cc:740] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.707962 3957 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.708218 3957 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.708309 3955 consensus_queue.cc:260] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.708619 3957 ts_tablet_manager.cc:1434] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251026 08:52:44.708829 3957 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: Bootstrap starting.
I20251026 08:52:44.710366 3957 tablet_bootstrap.cc:654] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.711606 3955 ts_tablet_manager.cc:1434] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 08:52:44.712071 3962 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.712229 3962 ts_tablet_manager.cc:1403] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20251026 08:52:44.712930 3962 raft_consensus.cc:359] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.713127 3962 raft_consensus.cc:385] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.713239 3962 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.713471 3962 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.713913 3955 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.713951 3962 ts_tablet_manager.cc:1434] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:52:44.714592 3962 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.716284 3955 tablet_bootstrap.cc:654] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.716925 3962 tablet_bootstrap.cc:654] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.718182 3957 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: No bootstrap required, opened a new log
I20251026 08:52:44.718338 3957 ts_tablet_manager.cc:1403] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20251026 08:52:44.718938 3957 raft_consensus.cc:359] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.719117 3957 raft_consensus.cc:385] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.719209 3957 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.719432 3957 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.719825 3957 ts_tablet_manager.cc:1434] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:52:44.723014 3962 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.723141 3962 ts_tablet_manager.cc:1403] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20251026 08:52:44.723634 3962 raft_consensus.cc:359] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.723757 3962 raft_consensus.cc:385] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.723845 3962 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.724025 3962 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.724472 3962 ts_tablet_manager.cc:1434] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:52:44.724735 3962 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Bootstrap starting.
I20251026 08:52:44.726039 3962 tablet_bootstrap.cc:654] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.728112 3955 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.728294 3955 ts_tablet_manager.cc:1403] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20251026 08:52:44.728538 3962 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: No bootstrap required, opened a new log
I20251026 08:52:44.728684 3962 ts_tablet_manager.cc:1403] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:52:44.729111 3955 raft_consensus.cc:359] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.729312 3955 raft_consensus.cc:385] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.729429 3955 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.729389 3962 raft_consensus.cc:359] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.729579 3962 raft_consensus.cc:385] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.729664 3962 raft_consensus.cc:740] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.729688 3955 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.729941 3962 consensus_queue.cc:260] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.730248 3955 ts_tablet_manager.cc:1434] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:52:44.730525 3955 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: Bootstrap starting.
I20251026 08:52:44.730527 3962 ts_tablet_manager.cc:1434] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:52:44.731813 3955 tablet_bootstrap.cc:654] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: Neither blocks nor log segments found. Creating new log.
I20251026 08:52:44.733809 3955 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: No bootstrap required, opened a new log
I20251026 08:52:44.733985 3955 ts_tablet_manager.cc:1403] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251026 08:52:44.734607 3955 raft_consensus.cc:359] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.734755 3955 raft_consensus.cc:385] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:52:44.734817 3955 raft_consensus.cc:740] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Initialized, Role: FOLLOWER
I20251026 08:52:44.735015 3955 consensus_queue.cc:260] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.735407 3955 ts_tablet_manager.cc:1434] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:52:44.775488 3993 raft_consensus.cc:493] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.775630 3993 raft_consensus.cc:515] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.776072 3993 leader_election.cc:290] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.776503 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.776523 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.776719 3910 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.776717 3760 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.777151 3801 leader_election.cc:304] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.777416 3993 raft_consensus.cc:2804] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.777523 3993 raft_consensus.cc:493] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.777632 3993 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.779479 3993 raft_consensus.cc:515] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.779965 3993 leader_election.cc:290] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.780386 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.780393 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.780553 3910 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.780596 3760 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.782174 3760 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.782542 3802 leader_election.cc:304] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.782696 3910 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.782749 3993 raft_consensus.cc:2804] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.782871 3993 raft_consensus.cc:697] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:44.783085 3993 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.783950 3968 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.784127 3968 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.784678 3968 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.785044 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.785251 3760 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.785290 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.785460 3910 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.785548 3802 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.785573 3656 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.785813 3968 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.785925 3968 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.786020 3968 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.787714 3968 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.788250 3968 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.788600 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.788707 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.788743 3760 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.788856 3910 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.790330 3760 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.790584 3910 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.790622 3802 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.790812 3968 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.790925 3968 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:44.791118 3968 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.792984 3656 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.798013 3960 raft_consensus.cc:493] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.798190 3960 raft_consensus.cc:515] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.798658 3960 leader_election.cc:290] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.799048 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.799149 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.799209 3760 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 0.
I20251026 08:52:44.799350 3836 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 0.
I20251026 08:52:44.799702 3876 leader_election.cc:304] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.800009 3960 raft_consensus.cc:2804] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.800114 3960 raft_consensus.cc:493] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.800215 3960 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.801693 3960 raft_consensus.cc:515] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.801844 3981 raft_consensus.cc:493] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.801970 3981 raft_consensus.cc:515] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.802248 3960 leader_election.cc:290] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.802556 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.802688 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:44.802760 3760 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.802865 3836 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.802991 3761 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.803185 3761 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 0.
I20251026 08:52:44.802724 3981 leader_election.cc:290] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.803890 3876 leader_election.cc:304] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.803259 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.804085 3835 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 0.
I20251026 08:52:44.804118 3981 raft_consensus.cc:2804] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.804657 3981 raft_consensus.cc:493] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.804754 3981 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.805294 3836 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 1.
I20251026 08:52:44.805577 3875 leader_election.cc:304] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:44.805771 3960 raft_consensus.cc:2804] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.805895 3960 raft_consensus.cc:697] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:44.806142 3960 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.806149 3760 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 1.
I20251026 08:52:44.807168 3981 raft_consensus.cc:515] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.808012 3981 leader_election.cc:290] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.808149 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.808292 3760 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.808535 3656 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:44.809226 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:44.809422 3836 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.810328 3760 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 1.
I20251026 08:52:44.810701 3876 leader_election.cc:304] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.810957 3981 raft_consensus.cc:2804] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.811091 3981 raft_consensus.cc:697] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:44.811313 3981 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.811462 3836 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 1.
I20251026 08:52:44.813223 3656 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:44.836418 3973 raft_consensus.cc:493] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.836560 3973 raft_consensus.cc:515] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.837050 3973 leader_election.cc:290] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.837325 3968 raft_consensus.cc:493] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.837534 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.837502 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.837498 3968 raft_consensus.cc:515] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.837716 3836 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.837721 3910 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.838112 3968 leader_election.cc:290] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.838140 3726 leader_election.cc:304] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.838366 3973 raft_consensus.cc:2804] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.838941 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.838989 3973 raft_consensus.cc:493] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.839027 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.839139 3973 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.839200 3760 raft_consensus.cc:2393] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96cd069722b34c1eab8ae9a2f953dd7a in current term 1: Already voted for candidate 53f35e5629a94c8f81ad198ebc4df6ae in this term.
I20251026 08:52:44.839550 3802 leader_election.cc:304] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a; no voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:44.839774 3968 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.841131 3973 raft_consensus.cc:515] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.841542 3973 leader_election.cc:290] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.841550 3968 raft_consensus.cc:2749] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 08:52:44.841979 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:44.842202 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.842317 3910 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.843710 3910 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 1.
I20251026 08:52:44.843991 3726 leader_election.cc:304] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.844182 3973 raft_consensus.cc:2804] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.844306 3973 raft_consensus.cc:697] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:44.844480 3973 consensus_queue.cc:237] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.846622 3656 catalog_manager.cc:5649] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.847522 3836 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.848050 3968 raft_consensus.cc:493] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.848148 3968 raft_consensus.cc:515] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.848882 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.848876 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.849072 3910 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.849169 3760 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 0.
I20251026 08:52:44.849278 3836 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 1.
I20251026 08:52:44.849421 3802 leader_election.cc:304] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.849586 3968 leader_election.cc:290] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.849607 3993 raft_consensus.cc:2804] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.849684 3993 raft_consensus.cc:493] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.849735 3993 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.851079 3993 raft_consensus.cc:515] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.851435 3993 leader_election.cc:290] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.851747 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:44.851774 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.851876 3760 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.851886 3910 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.853263 3760 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.853338 3910 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 1.
I20251026 08:52:44.853567 3802 leader_election.cc:304] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.853730 3993 raft_consensus.cc:2804] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.853833 3993 raft_consensus.cc:697] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:44.854003 3993 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.855859 3656 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.866721 3981 raft_consensus.cc:493] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.866822 3981 raft_consensus.cc:515] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.867193 3981 leader_election.cc:290] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:44.867578 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.867578 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1" is_pre_election: true
I20251026 08:52:44.867780 3836 raft_consensus.cc:2393] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 53f35e5629a94c8f81ad198ebc4df6ae in current term 1: Already voted for candidate e7be295d3ba247a1a4475fa020d1ade1 in this term.
I20251026 08:52:44.868049 3875 leader_election.cc:304] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae; no voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:44.868206 3981 raft_consensus.cc:3060] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.869514 3981 raft_consensus.cc:2749] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251026 08:52:44.872841 3760 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.874703 3993 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:44.878667 3959 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:44.934536 3973 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:52:44.934720 3973 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.935292 3973 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.935801 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" is_pre_election: true
I20251026 08:52:44.935801 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" is_pre_election: true
I20251026 08:52:44.936070 3910 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.936070 3836 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 0.
I20251026 08:52:44.936520 3726 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.936729 3973 raft_consensus.cc:2804] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:52:44.936806 3973 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:52:44.936882 3973 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.938635 3973 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.939091 3973 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:44.939510 3836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:44.939544 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:44.939661 3910 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.939658 3836 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:52:44.941241 3910 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 1.
I20251026 08:52:44.941380 3836 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 1.
I20251026 08:52:44.941531 3726 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:44.941740 3973 raft_consensus.cc:2804] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:52:44.941866 3973 raft_consensus.cc:697] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:44.942122 3973 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:44.944222 3657 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:44.989980 3981 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:44.993506 3960 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.046257 3959 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.059399 3760 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.059993 3761 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.060449 3836 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.061538 3758 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.066781 3979 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.066880 3960 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.070214 3908 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.071187 3836 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.071194 3979 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251026 08:52:45.079704 3959 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.081682 3960 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.083302 3968 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.093318 3907 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.093894 3993 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.105195 3906 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.105885 3959 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.114327 3755 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:52:45.116165 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:45.116426 3968 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.199556 3760 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251026 08:52:45.200107 3875 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.66:37913, user_credentials={real_user=slave}} blocked reactor thread for 67611.3us
I20251026 08:52:45.202131 3994 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.203217 3968 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.129598 4038 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.074s
I20251026 08:52:45.212360 3994 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.250301 3994 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.274515 3994 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:52:45.355695 4051 log.cc:927] Time spent T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
W20251026 08:52:45.395648 4072 log.cc:927] Time spent T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20251026 08:52:45.423877 3994 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.438325 3994 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:52:45.449333 3994 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:52:45.459955 4066 log.cc:927] Time spent T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20251026 08:52:45.477245 3994 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.014s
W20251026 08:52:45.497395 3802 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.65:39515, user_credentials={real_user=slave}} blocked reactor thread for 103558us
W20251026 08:52:45.525606 4076 log.cc:927] Time spent T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20251026 08:52:45.559224 3834 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:45.559392 3834 raft_consensus.cc:606] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:45.559988 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:52:45.560191 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 08:52:45.561331 3726 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.103.67:33171, user_credentials={real_user=slave}} blocked reactor thread for 68660.4us
W20251026 08:52:45.666662 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.697741 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.719274 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.746603 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.761060 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.767802 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.786202 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.778352 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.789193 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.812332 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.816063 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.816116 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.817703 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.834055 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.834408 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.837764 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.842211 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.849581 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.851893 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.852648 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.853283 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.857625 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.861796 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.862704 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.863611 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.865691 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.872839 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.872854 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.876008 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.876945 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.882378 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.885654 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.885759 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.888923 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.895102 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.899314 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.899408 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.903532 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.905670 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.914093 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.914208 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.917331 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.919463 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.926538 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.929690 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.930619 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.931550 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.942850 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.946007 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.948361 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.948361 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.959702 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.962910 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.963781 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.965732 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.975742 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.979794 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.980666 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.981489 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.996773 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.996891 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.996963 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:45.999192 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.014547 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.016726 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.017689 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.018661 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.037215 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.037484 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.037966 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.041296 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.058954 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.058810 3816 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.058810 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.061554 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.082705 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.082886 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.083729 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.083838 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.105988 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.107105 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.107244 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
W20251026 08:52:46.108175 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Service unavailable: leader transfer in progress
I20251026 08:52:46.112246 4079 raft_consensus.cc:993] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:46.112334 4079 raft_consensus.cc:1081] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:46.112876 3910 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:52:46.113009 3910 raft_consensus.cc:493] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:46.113090 3910 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:46.115074 3910 raft_consensus.cc:515] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:46.115661 3910 leader_election.cc:290] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:46.116039 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:46.116042 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:46.116194 3832 raft_consensus.cc:3055] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:46.116222 3760 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:46.116261 3832 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:46.116500 3832 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:46.116755 3832 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:46.118207 3760 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:46.118569 3876 leader_election.cc:304] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:46.118671 3832 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:46.118844 4058 raft_consensus.cc:2804] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:46.119012 4058 raft_consensus.cc:697] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:46.119299 4058 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:46.121285 3656 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 1 to 2, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
W20251026 08:52:46.129662 3808 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Illegal state: replica 96cd069722b34c1eab8ae9a2f953dd7a is not leader of this config: current role FOLLOWER
W20251026 08:52:46.129696 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Illegal state: replica 96cd069722b34c1eab8ae9a2f953dd7a is not leader of this config: current role FOLLOWER
W20251026 08:52:46.134034 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Illegal state: replica 96cd069722b34c1eab8ae9a2f953dd7a is not leader of this config: current role FOLLOWER
W20251026 08:52:46.135082 3807 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:60870: Illegal state: replica 96cd069722b34c1eab8ae9a2f953dd7a is not leader of this config: current role FOLLOWER
W20251026 08:52:46.154609 3733 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51968: Illegal state: replica e7be295d3ba247a1a4475fa020d1ade1 is not leader of this config: current role FOLLOWER
W20251026 08:52:46.155588 3733 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51968: Illegal state: replica e7be295d3ba247a1a4475fa020d1ade1 is not leader of this config: current role FOLLOWER
W20251026 08:52:46.159874 3733 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51968: Illegal state: replica e7be295d3ba247a1a4475fa020d1ade1 is not leader of this config: current role FOLLOWER
W20251026 08:52:46.163089 3733 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:51968: Illegal state: replica e7be295d3ba247a1a4475fa020d1ade1 is not leader of this config: current role FOLLOWER
I20251026 08:52:46.181883 3832 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20251026 08:52:46.181900 3760 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20251026 08:52:46.182354 3960 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:52:46.182505 4058 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:52:46.186338 4093 mvcc.cc:204] Tried to move back new op lower bound from 7214976066277003264 to 7214976066026524672. Current Snapshot: MvccSnapshot[applied={T|T < 7214976066277003264 or (T in {7214976066277003264})}]
I20251026 08:52:46.704769 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:46.707355 4105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:46.708238 4106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:46.714141 413 server_base.cc:1047] running on GCE node
W20251026 08:52:46.714781 4109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:46.715032 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:46.715102 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:46.715145 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468766715142 us; error 0 us; skew 500 ppm
I20251026 08:52:46.715270 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:46.716349 413 webserver.cc:492] Webserver started at http://127.0.103.68:44421/ using document root <none> and password file <none>
I20251026 08:52:46.716576 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:46.716642 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:46.716766 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:46.717324 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "381d5756da8c4fa3ab5663d92bf76739"
format_stamp: "Formatted at 2025-10-26 08:52:46 on dist-test-slave-rd83"
I20251026 08:52:46.719790 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 08:52:46.721109 4113 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.721412 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:52:46.721498 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "381d5756da8c4fa3ab5663d92bf76739"
format_stamp: "Formatted at 2025-10-26 08:52:46 on dist-test-slave-rd83"
I20251026 08:52:46.721580 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:46.735740 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:46.736191 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:46.736663 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:46.737874 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:46.737924 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.737980 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:46.738013 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.751786 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:33881
I20251026 08:52:46.751822 4175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:33881 every 8 connection(s)
I20251026 08:52:46.753290 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:46.755741 4180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:46.756878 4181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:46.757936 4183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:46.760298 4176 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39299
I20251026 08:52:46.760416 4176 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:46.760695 4176 heartbeater.cc:507] Master 127.0.103.126:39299 requested a full tablet report, sending...
I20251026 08:52:46.761490 3656 ts_manager.cc:194] Registered new tserver with Master: 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881)
I20251026 08:52:46.762681 3656 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49182
I20251026 08:52:46.763145 413 server_base.cc:1047] running on GCE node
I20251026 08:52:46.763371 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:46.763449 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:46.763489 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468766763486 us; error 0 us; skew 500 ppm
I20251026 08:52:46.763617 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:46.764671 413 webserver.cc:492] Webserver started at http://127.0.103.69:37821/ using document root <none> and password file <none>
I20251026 08:52:46.764827 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:46.764868 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:46.764943 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:46.765412 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-4-root/instance:
uuid: "49e9ac1b52f24abf8690af5a6924646c"
format_stamp: "Formatted at 2025-10-26 08:52:46 on dist-test-slave-rd83"
I20251026 08:52:46.767612 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:46.768896 4188 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.769304 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:46.769384 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-4-root
uuid: "49e9ac1b52f24abf8690af5a6924646c"
format_stamp: "Formatted at 2025-10-26 08:52:46 on dist-test-slave-rd83"
I20251026 08:52:46.769455 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-4-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-4-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-4-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:46.781675 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:46.782032 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:46.782900 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:46.783514 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:46.783556 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.783610 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:46.783646 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.796072 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.69:34723
I20251026 08:52:46.796098 4250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.69:34723 every 8 connection(s)
I20251026 08:52:46.797822 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:52:46.800205 4255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:52:46.800738 4254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:46.801280 413 server_base.cc:1047] running on GCE node
W20251026 08:52:46.801510 4257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:52:46.801874 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:52:46.801946 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:52:46.801980 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468766801976 us; error 0 us; skew 500 ppm
I20251026 08:52:46.802084 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:52:46.803138 413 webserver.cc:492] Webserver started at http://127.0.103.70:39581/ using document root <none> and password file <none>
I20251026 08:52:46.803324 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:52:46.803380 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:52:46.803465 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:52:46.803602 4251 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39299
I20251026 08:52:46.803704 4251 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:46.803985 4251 heartbeater.cc:507] Master 127.0.103.126:39299 requested a full tablet report, sending...
I20251026 08:52:46.804081 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-5-root/instance:
uuid: "b983c9a7c2924030a396232a4ad46572"
format_stamp: "Formatted at 2025-10-26 08:52:46 on dist-test-slave-rd83"
I20251026 08:52:46.804629 3656 ts_manager.cc:194] Registered new tserver with Master: 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:52:46.805238 3656 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49198
I20251026 08:52:46.806187 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:52:46.807515 4262 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.807854 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:46.807922 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-5-root
uuid: "b983c9a7c2924030a396232a4ad46572"
format_stamp: "Formatted at 2025-10-26 08:52:46 on dist-test-slave-rd83"
I20251026 08:52:46.807983 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-5-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-5-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-5-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:52:46.829212 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:52:46.829618 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:52:46.830503 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:52:46.831147 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:52:46.831190 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.831233 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:52:46.831274 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:52:46.843950 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.70:46541
I20251026 08:52:46.844035 4324 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.70:46541 every 8 connection(s)
I20251026 08:52:46.849810 4325 heartbeater.cc:344] Connected to a master server at 127.0.103.126:39299
I20251026 08:52:46.849900 4325 heartbeater.cc:461] Registering TS with master...
I20251026 08:52:46.850236 4325 heartbeater.cc:507] Master 127.0.103.126:39299 requested a full tablet report, sending...
I20251026 08:52:46.850863 3656 ts_manager.cc:194] Registered new tserver with Master: b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:52:46.851531 3656 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:49202
I20251026 08:52:47.330999 3909 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 2.26, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:47.332909 3832 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:47.333354 4331 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:47.333755 3758 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:47.334141 4331 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:47.335281 3979 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.336129 3832 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.336323 3758 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.27: config changed from index -1 to 27, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.340667 3909 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
W20251026 08:52:47.340799 3876 consensus_peers.cc:597] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b2c4bd03bfd4efd8aa306d8557fad3e. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:47.340898 3657 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index -1 to 27, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.343353 3758 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:47.343886 3832 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:47.344041 3979 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:47.344316 4334 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:47.346377 4335 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:47.347409 3758 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:47.350296 3832 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
W20251026 08:52:47.353111 3873 consensus_peers.cc:597] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1e736aea2b54b378ffbe821cb9e42c4. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:47.361006 3650 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:47.363394 3758 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:47.366264 3832 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:47.366499 3909 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:47.366789 4088 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:47.367403 4088 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:47.369259 4088 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:47.373214 3656 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:47.378718 3832 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:47.379640 3910 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:47.383188 3909 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
W20251026 08:52:47.384423 3726 consensus_peers.cc:597] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0409b48759d34a6fa9c86d88b809e093. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:47.384871 3758 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251026 08:52:47.385227 4334 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251026 08:52:47.385349 3832 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251026 08:52:47.385751 4334 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251026 08:52:47.387410 4335 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:47.387954 3758 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:47.388156 3832 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
W20251026 08:52:47.390019 3875 consensus_peers.cc:597] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85fe4797e56f4170876f76414e23628f. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:47.391728 3656 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index -1 to 26, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:47.402057 3832 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:47.403764 3909 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251026 08:52:47.404163 4348 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251026 08:52:47.405853 4038 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.406489 3909 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.409526 3758 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251026 08:52:47.410514 3758 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:47.410830 3657 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index -1 to 26, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:47.411881 4038 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20251026 08:52:47.412883 3909 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251026 08:52:47.413447 3832 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
I20251026 08:52:47.414005 4343 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20251026 08:52:47.415261 3802 consensus_peers.cc:597] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0a49b69e0a94ae9827e91c43006c2d5. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:47.414433 4343 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.001s
I20251026 08:52:47.415416 3758 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:47.416123 4088 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:47.416970 3832 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
W20251026 08:52:47.419131 3724 consensus_peers.cc:597] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5f200f20b1741b4a2c6bc379b2ae8f5. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:47.418833 3909 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:47.420632 3656 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index -1 to 26, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:47.560477 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:47.578585 3909 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "85fe4797e56f4170876f76414e23628f"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:47.578744 3909 raft_consensus.cc:606] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:47.579111 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:52:47.579325 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:47.718268 4351 ts_tablet_manager.cc:933] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:47.718813 4351 tablet_copy_client.cc:323] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:47.724264 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e from peer 381d5756da8c4fa3ab5663d92bf76739 ({username='slave'} at 127.0.0.1:47624)
I20251026 08:52:47.724417 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet 0b2c4bd03bfd4efd8aa306d8557fad3e from peer 381d5756da8c4fa3ab5663d92bf76739 at {username='slave'} at 127.0.0.1:47624: session id = 381d5756da8c4fa3ab5663d92bf76739-0b2c4bd03bfd4efd8aa306d8557fad3e
I20251026 08:52:47.726612 3920 tablet_copy_source_session.cc:215] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:47.727701 4351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:47.733060 4351 tablet_copy_client.cc:806] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:47.733260 4351 tablet_copy_client.cc:670] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:47.734557 4352 tablet_copy_client.cc:1025] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:47.763988 4176 heartbeater.cc:499] Master 127.0.103.126:39299 was elected leader, sending a full tablet report...
I20251026 08:52:47.771117 4354 ts_tablet_manager.cc:933] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:47.771711 4354 tablet_copy_client.cc:323] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:47.777288 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet 85fe4797e56f4170876f76414e23628f from peer 49e9ac1b52f24abf8690af5a6924646c ({username='slave'} at 127.0.0.1:47636)
I20251026 08:52:47.777422 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet 85fe4797e56f4170876f76414e23628f from peer 49e9ac1b52f24abf8690af5a6924646c at {username='slave'} at 127.0.0.1:47636: session id = 49e9ac1b52f24abf8690af5a6924646c-85fe4797e56f4170876f76414e23628f
I20251026 08:52:47.779485 3920 tablet_copy_source_session.cc:215] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:47.780568 4354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe4797e56f4170876f76414e23628f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:47.785904 4354 tablet_copy_client.cc:806] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:47.786157 4354 tablet_copy_client.cc:670] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:47.787503 4355 tablet_copy_client.cc:1025] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:47.806334 4251 heartbeater.cc:499] Master 127.0.103.126:39299 was elected leader, sending a full tablet report...
I20251026 08:52:47.814689 4357 ts_tablet_manager.cc:933] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Initiating tablet copy from peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:47.815438 4357 tablet_copy_client.cc:323] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:39515
I20251026 08:52:47.831593 3771 tablet_copy_service.cc:140] P e7be295d3ba247a1a4475fa020d1ade1: Received BeginTabletCopySession request for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 from peer b983c9a7c2924030a396232a4ad46572 ({username='slave'} at 127.0.0.1:46300)
I20251026 08:52:47.831743 3771 tablet_copy_service.cc:161] P e7be295d3ba247a1a4475fa020d1ade1: Beginning new tablet copy session on tablet c5f200f20b1741b4a2c6bc379b2ae8f5 from peer b983c9a7c2924030a396232a4ad46572 at {username='slave'} at 127.0.0.1:46300: session id = b983c9a7c2924030a396232a4ad46572-c5f200f20b1741b4a2c6bc379b2ae8f5
I20251026 08:52:47.834054 3771 tablet_copy_source_session.cc:215] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:47.835378 4357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f200f20b1741b4a2c6bc379b2ae8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:47.839991 4357 tablet_copy_client.cc:806] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:47.840256 4357 tablet_copy_client.cc:670] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:47.841624 4358 tablet_copy_client.cc:1025] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:47.852474 4325 heartbeater.cc:499] Master 127.0.103.126:39299 was elected leader, sending a full tablet report...
I20251026 08:52:47.916157 4361 ts_tablet_manager.cc:933] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Initiating tablet copy from peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:47.916747 4361 tablet_copy_client.cc:323] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:39515
I20251026 08:52:47.921603 3771 tablet_copy_service.cc:140] P e7be295d3ba247a1a4475fa020d1ade1: Received BeginTabletCopySession request for tablet 0409b48759d34a6fa9c86d88b809e093 from peer 49e9ac1b52f24abf8690af5a6924646c ({username='slave'} at 127.0.0.1:46316)
I20251026 08:52:47.921759 3771 tablet_copy_service.cc:161] P e7be295d3ba247a1a4475fa020d1ade1: Beginning new tablet copy session on tablet 0409b48759d34a6fa9c86d88b809e093 from peer 49e9ac1b52f24abf8690af5a6924646c at {username='slave'} at 127.0.0.1:46316: session id = 49e9ac1b52f24abf8690af5a6924646c-0409b48759d34a6fa9c86d88b809e093
I20251026 08:52:47.923974 3771 tablet_copy_source_session.cc:215] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:47.924888 4361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0409b48759d34a6fa9c86d88b809e093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:47.928807 4361 tablet_copy_client.cc:806] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:47.929045 4361 tablet_copy_client.cc:670] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:47.930200 4362 tablet_copy_client.cc:1025] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:47.942160 4363 ts_tablet_manager.cc:933] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: Initiating tablet copy from peer 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:47.942735 4363 tablet_copy_client.cc:323] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:37913
I20251026 08:52:47.947919 3846 tablet_copy_service.cc:140] P 96cd069722b34c1eab8ae9a2f953dd7a: Received BeginTabletCopySession request for tablet d0a49b69e0a94ae9827e91c43006c2d5 from peer 381d5756da8c4fa3ab5663d92bf76739 ({username='slave'} at 127.0.0.1:42606)
I20251026 08:52:47.948024 3846 tablet_copy_service.cc:161] P 96cd069722b34c1eab8ae9a2f953dd7a: Beginning new tablet copy session on tablet d0a49b69e0a94ae9827e91c43006c2d5 from peer 381d5756da8c4fa3ab5663d92bf76739 at {username='slave'} at 127.0.0.1:42606: session id = 381d5756da8c4fa3ab5663d92bf76739-d0a49b69e0a94ae9827e91c43006c2d5
I20251026 08:52:47.949910 3846 tablet_copy_source_session.cc:215] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:47.950863 4363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a49b69e0a94ae9827e91c43006c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:47.954707 4366 ts_tablet_manager.cc:933] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:47.955325 4366 tablet_copy_client.cc:323] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:47.955662 4363 tablet_copy_client.cc:806] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:47.955901 4363 tablet_copy_client.cc:670] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:47.957511 4364 tablet_copy_client.cc:1025] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:47.962621 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet c1e736aea2b54b378ffbe821cb9e42c4 from peer b983c9a7c2924030a396232a4ad46572 ({username='slave'} at 127.0.0.1:47646)
I20251026 08:52:47.962738 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet c1e736aea2b54b378ffbe821cb9e42c4 from peer b983c9a7c2924030a396232a4ad46572 at {username='slave'} at 127.0.0.1:47646: session id = b983c9a7c2924030a396232a4ad46572-c1e736aea2b54b378ffbe821cb9e42c4
I20251026 08:52:47.964903 3920 tablet_copy_source_session.cc:215] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:47.965799 4366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e736aea2b54b378ffbe821cb9e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:47.971004 4366 tablet_copy_client.cc:806] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:47.971271 4366 tablet_copy_client.cc:670] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:47.972734 4367 tablet_copy_client.cc:1025] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:47.990463 4368 raft_consensus.cc:993] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:47.990638 4368 raft_consensus.cc:1081] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:47.991135 3758 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "85fe4797e56f4170876f76414e23628f"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51954
I20251026 08:52:47.991307 3758 raft_consensus.cc:493] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:47.991449 3758 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:47.994848 3758 raft_consensus.cc:515] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:47.995499 3758 leader_election.cc:290] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 2 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:47.995774 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:47.995738 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:47.995927 3835 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:47.995993 3910 raft_consensus.cc:3055] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:47.996114 3910 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:47.996389 3910 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:47.996748 3910 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:47.997781 3835 raft_consensus.cc:2468] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 2.
I20251026 08:52:47.998241 3726 leader_election.cc:304] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:47.998489 4343 raft_consensus.cc:2804] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:47.998613 4343 raft_consensus.cc:697] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:47.998867 4343 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:48.000165 3910 raft_consensus.cc:2468] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 2.
I20251026 08:52:48.001245 3650 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 1 to 2, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 08:52:48.406544 3726 consensus_peers.cc:597] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:48.419593 3832 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:48.420049 4346 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:48.431326 3910 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:48.431725 4376 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:49.579702 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:49.582053 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:52:49.582249 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:49.734900 4351 tablet_copy_client.cc:538] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:49.738070 4351 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap starting.
I20251026 08:52:49.771936 4351 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:49.772403 4351 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap complete.
I20251026 08:52:49.772637 4351 ts_tablet_manager.cc:1403] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.008s
I20251026 08:52:49.773339 4351 raft_consensus.cc:359] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:49.773517 4351 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Initialized, Role: LEARNER
I20251026 08:52:49.773701 4351 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:49.774394 4351 ts_tablet_manager.cc:1434] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:52:49.774973 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session 381d5756da8c4fa3ab5663d92bf76739-0b2c4bd03bfd4efd8aa306d8557fad3e received from {username='slave'} at 127.0.0.1:47624
I20251026 08:52:49.775079 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session 381d5756da8c4fa3ab5663d92bf76739-0b2c4bd03bfd4efd8aa306d8557fad3e on tablet 0b2c4bd03bfd4efd8aa306d8557fad3e with peer 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:52:49.787822 4354 tablet_copy_client.cc:538] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:49.790652 4354 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap starting.
I20251026 08:52:49.824002 4354 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:49.824371 4354 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap complete.
I20251026 08:52:49.824575 4354 ts_tablet_manager.cc:1403] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.002s
I20251026 08:52:49.825279 4354 raft_consensus.cc:359] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:49.825475 4354 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Initialized, Role: LEARNER
I20251026 08:52:49.825706 4354 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:49.826371 4151 raft_consensus.cc:1217] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.26->[2.27-2.27] Dedup: 2.27->[]
I20251026 08:52:49.826418 4354 ts_tablet_manager.cc:1434] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20251026 08:52:49.827827 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session 49e9ac1b52f24abf8690af5a6924646c-85fe4797e56f4170876f76414e23628f received from {username='slave'} at 127.0.0.1:47636
I20251026 08:52:49.827943 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session 49e9ac1b52f24abf8690af5a6924646c-85fe4797e56f4170876f76414e23628f on tablet 85fe4797e56f4170876f76414e23628f with peer 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:52:49.841912 4357 tablet_copy_client.cc:538] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:49.844797 4357 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Bootstrap starting.
I20251026 08:52:49.874310 4357 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:49.874663 4357 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Bootstrap complete.
I20251026 08:52:49.874866 4357 ts_tablet_manager.cc:1403] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Time spent bootstrapping tablet: real 0.030s user 0.029s sys 0.002s
I20251026 08:52:49.875380 4357 raft_consensus.cc:359] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:49.875557 4357 raft_consensus.cc:740] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Initialized, Role: LEARNER
I20251026 08:52:49.875743 4357 consensus_queue.cc:260] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:49.876320 4357 ts_tablet_manager.cc:1434] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:52:49.876871 3771 tablet_copy_service.cc:342] P e7be295d3ba247a1a4475fa020d1ade1: Request end of tablet copy session b983c9a7c2924030a396232a4ad46572-c5f200f20b1741b4a2c6bc379b2ae8f5 received from {username='slave'} at 127.0.0.1:46300
I20251026 08:52:49.876986 3771 tablet_copy_service.cc:434] P e7be295d3ba247a1a4475fa020d1ade1: ending tablet copy session b983c9a7c2924030a396232a4ad46572-c5f200f20b1741b4a2c6bc379b2ae8f5 on tablet c5f200f20b1741b4a2c6bc379b2ae8f5 with peer b983c9a7c2924030a396232a4ad46572
I20251026 08:52:49.930536 4361 tablet_copy_client.cc:538] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:49.933106 4361 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap starting.
I20251026 08:52:49.957850 4363 tablet_copy_client.cc:538] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:49.960650 4363 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap starting.
I20251026 08:52:49.966125 4361 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:49.966514 4361 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap complete.
I20251026 08:52:49.966724 4361 ts_tablet_manager.cc:1403] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Time spent bootstrapping tablet: real 0.034s user 0.033s sys 0.000s
I20251026 08:52:49.967518 4361 raft_consensus.cc:359] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:49.967736 4361 raft_consensus.cc:740] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Initialized, Role: LEARNER
I20251026 08:52:49.967944 4361 consensus_queue.cc:260] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:49.968443 4361 ts_tablet_manager.cc:1434] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:52:49.968993 3771 tablet_copy_service.cc:342] P e7be295d3ba247a1a4475fa020d1ade1: Request end of tablet copy session 49e9ac1b52f24abf8690af5a6924646c-0409b48759d34a6fa9c86d88b809e093 received from {username='slave'} at 127.0.0.1:46316
I20251026 08:52:49.969095 3771 tablet_copy_service.cc:434] P e7be295d3ba247a1a4475fa020d1ade1: ending tablet copy session 49e9ac1b52f24abf8690af5a6924646c-0409b48759d34a6fa9c86d88b809e093 on tablet 0409b48759d34a6fa9c86d88b809e093 with peer 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:52:49.973039 4366 tablet_copy_client.cc:538] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:49.975700 4366 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: Bootstrap starting.
I20251026 08:52:49.995543 4363 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap replayed 1/1 log segments. Stats: ops{read=26 overwritten=0 applied=26 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:49.995914 4363 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap complete.
I20251026 08:52:49.996107 4363 ts_tablet_manager.cc:1403] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.007s
I20251026 08:52:49.996662 4363 raft_consensus.cc:359] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:49.996863 4363 raft_consensus.cc:740] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Initialized, Role: LEARNER
I20251026 08:52:49.997133 4363 consensus_queue.cc:260] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:49.997591 4363 ts_tablet_manager.cc:1434] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:52:49.998207 3846 tablet_copy_service.cc:342] P 96cd069722b34c1eab8ae9a2f953dd7a: Request end of tablet copy session 381d5756da8c4fa3ab5663d92bf76739-d0a49b69e0a94ae9827e91c43006c2d5 received from {username='slave'} at 127.0.0.1:42606
I20251026 08:52:49.998307 3846 tablet_copy_service.cc:434] P 96cd069722b34c1eab8ae9a2f953dd7a: ending tablet copy session 381d5756da8c4fa3ab5663d92bf76739-d0a49b69e0a94ae9827e91c43006c2d5 on tablet d0a49b69e0a94ae9827e91c43006c2d5 with peer 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:52:50.011062 4366 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:50.011420 4366 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: Bootstrap complete.
I20251026 08:52:50.011632 4366 ts_tablet_manager.cc:1403] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: Time spent bootstrapping tablet: real 0.036s user 0.028s sys 0.007s
I20251026 08:52:50.012209 4366 raft_consensus.cc:359] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:50.012351 4366 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Initialized, Role: LEARNER
I20251026 08:52:50.012485 4366 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:50.012868 4366 ts_tablet_manager.cc:1434] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:52:50.013324 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session b983c9a7c2924030a396232a4ad46572-c1e736aea2b54b378ffbe821cb9e42c4 received from {username='slave'} at 127.0.0.1:47646
I20251026 08:52:50.013417 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session b983c9a7c2924030a396232a4ad46572-c1e736aea2b54b378ffbe821cb9e42c4 on tablet c1e736aea2b54b378ffbe821cb9e42c4 with peer b983c9a7c2924030a396232a4ad46572
I20251026 08:52:50.106431 4151 raft_consensus.cc:1217] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20251026 08:52:50.202143 4226 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Advancing to term 2
I20251026 08:52:50.302856 4300 raft_consensus.cc:1217] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20251026 08:52:50.305279 4300 raft_consensus.cc:1217] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.26] Dedup: 1.26->[]
I20251026 08:52:50.351223 4392 raft_consensus.cc:1064] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: attempting to promote NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 to VOTER
I20251026 08:52:50.352017 4392 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:50.354313 4151 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:50.354454 3835 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:50.355048 4395 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.355294 3754 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:50.355667 4392 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.356104 4392 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.368372 3835 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.367782 4392 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.369671 3758 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.372937 3657 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 27 to 28, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.376468 4151 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 27 to 28, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.381548 3910 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:50.383074 4151 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:50.383404 3835 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:50.383451 4392 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.383777 4395 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.385380 4226 raft_consensus.cc:1217] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20251026 08:52:50.385638 4393 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.386401 3835 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.386893 4151 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.389319 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:50.391157 3657 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: config changed from index 28 to 29, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.394918 3748 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e with delete_type TABLET_DATA_TOMBSTONED (TS e7be295d3ba247a1a4475fa020d1ade1 not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:50.395591 4403 tablet_replica.cc:333] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:52:50.396310 4403 raft_consensus.cc:2243] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:50.396507 4403 raft_consensus.cc:2272] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:50.397337 4403 ts_tablet_manager.cc:1916] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:50.402472 4403 ts_tablet_manager.cc:1929] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20251026 08:52:50.402572 4403 log.cc:1199] T 0b2c4bd03bfd4efd8aa306d8557fad3e P e7be295d3ba247a1a4475fa020d1ade1: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/wals/0b2c4bd03bfd4efd8aa306d8557fad3e
I20251026 08:52:50.403478 3646 catalog_manager.cc:4985] TS e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): tablet 0b2c4bd03bfd4efd8aa306d8557fad3e (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:50.574309 4380 raft_consensus.cc:1064] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: attempting to promote NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 to VOTER
I20251026 08:52:50.574995 4380 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:50.576959 4151 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:50.577476 3758 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:50.577471 4405 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:50.577953 4404 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:50.577970 3910 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:50.578562 4405 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:50.592878 4380 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.595054 3758 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.597080 4151 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.597743 3650 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 26 to 27, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:50.601289 3907 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.603989 3835 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:50.605337 4151 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.605350 3907 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.605791 4405 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.605984 4404 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.608071 4380 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.608563 3907 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.609447 4151 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:50.611550 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet d0a49b69e0a94ae9827e91c43006c2d5 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:50.612560 3650 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 27 to 28, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:50.615792 3748 tablet_service.cc:1552] Processing DeleteTablet for tablet d0a49b69e0a94ae9827e91c43006c2d5 with delete_type TABLET_DATA_TOMBSTONED (TS e7be295d3ba247a1a4475fa020d1ade1 not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:50.616032 4403 tablet_replica.cc:333] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:52:50.616194 4403 raft_consensus.cc:2243] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:50.616371 4403 raft_consensus.cc:2272] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:50.617086 4403 ts_tablet_manager.cc:1916] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:50.622084 4403 ts_tablet_manager.cc:1929] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20251026 08:52:50.622193 4403 log.cc:1199] T d0a49b69e0a94ae9827e91c43006c2d5 P e7be295d3ba247a1a4475fa020d1ade1: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/wals/d0a49b69e0a94ae9827e91c43006c2d5
I20251026 08:52:50.622812 3646 catalog_manager.cc:4985] TS e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): tablet d0a49b69e0a94ae9827e91c43006c2d5 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:50.688921 4412 raft_consensus.cc:1064] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
I20251026 08:52:50.689762 4412 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:50.691809 4300 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:50.692267 4376 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:50.692683 3835 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:50.692817 3907 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:50.693100 4376 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:50.693401 4412 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:50.696612 4411 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index 26 to 27, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.697341 4300 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.698355 3910 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.701913 3835 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index 26 to 27, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.702885 3650 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 reported cstate change: config changed from index 26 to 27, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.707871 3758 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:50.709463 3835 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.709564 4300 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.709921 4417 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.710273 4412 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.712260 4376 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.713027 4300 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.714354 3835 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.715824 3646 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 with cas_config_opid_index 27: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:50.717120 3653 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 27 to 28, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:50.720485 3898 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 with delete_type TABLET_DATA_TOMBSTONED (TS 53f35e5629a94c8f81ad198ebc4df6ae not found in new config with opid_index 28) from {username='slave'} at 127.0.0.1:47574
I20251026 08:52:50.723672 4418 tablet_replica.cc:333] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:52:50.723847 4418 raft_consensus.cc:2243] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:50.724205 4418 raft_consensus.cc:2272] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:50.725066 4418 ts_tablet_manager.cc:1916] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:50.730173 4418 ts_tablet_manager.cc:1929] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.27
I20251026 08:52:50.730299 4418 log.cc:1199] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/wals/c5f200f20b1741b4a2c6bc379b2ae8f5
I20251026 08:52:50.730994 3645 catalog_manager.cc:4985] TS 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): tablet c5f200f20b1741b4a2c6bc379b2ae8f5 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:50.748936 4417 raft_consensus.cc:1064] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: attempting to promote NON_VOTER 49e9ac1b52f24abf8690af5a6924646c to VOTER
I20251026 08:52:50.749512 4417 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:50.751367 4226 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:50.751610 3835 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:50.751801 4417 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.752002 4376 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.752549 3910 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:50.753072 4417 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.755057 4376 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index 26 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.758694 4226 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.759349 3653 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 26 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:50.760433 3910 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.763316 3835 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index 26 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.769186 3758 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
W20251026 08:52:50.770036 4417 consensus_queue.cc:1175] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" responder_term: 2 status { last_received { term: 2 index: 28 } last_committed_idx: 28 last_received_current_leader { term: 2 index: 28 } }
I20251026 08:52:50.771180 4226 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:50.771191 3910 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:50.771584 4412 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.771838 4411 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.773940 4417 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.774340 3907 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.774755 4226 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.778270 3646 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 85fe4797e56f4170876f76414e23628f with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:50.779141 3653 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:50.782231 3822 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fe4797e56f4170876f76414e23628f with delete_type TABLET_DATA_TOMBSTONED (TS 96cd069722b34c1eab8ae9a2f953dd7a not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:60834
I20251026 08:52:50.782871 4422 tablet_replica.cc:333] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:52:50.783370 4422 raft_consensus.cc:2243] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:50.783682 4422 raft_consensus.cc:2272] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:50.784350 4422 ts_tablet_manager.cc:1916] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:50.788384 4422 ts_tablet_manager.cc:1929] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.28
I20251026 08:52:50.788473 4422 log.cc:1199] T 85fe4797e56f4170876f76414e23628f P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/wals/85fe4797e56f4170876f76414e23628f
I20251026 08:52:50.788985 3645 catalog_manager.cc:4985] TS 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): tablet 85fe4797e56f4170876f76414e23628f (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:50.902346 4423 raft_consensus.cc:1064] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
I20251026 08:52:50.904052 4423 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:50.906237 4300 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.906252 3835 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.906735 3754 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.906715 4395 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.907184 4425 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.907997 4425 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.910952 4395 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.911743 3754 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.914747 3835 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.918056 3657 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 27 to 28, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:50.919620 4412 raft_consensus.cc:1064] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: attempting to promote NON_VOTER 49e9ac1b52f24abf8690af5a6924646c to VOTER
I20251026 08:52:50.920367 4412 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:50.920624 4300 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.925158 3907 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:50.926496 4300 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251026 08:52:50.927313 4425 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.928350 3760 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251026 08:52:50.929052 4425 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.929901 4300 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.930210 3760 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.932068 4226 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.932487 4417 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.933604 3907 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.934195 4417 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.935349 3835 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:50.935134 3657 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.935775 4417 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:50.929612 4423 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:50.938212 4417 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.938531 3822 tablet_service.cc:1552] Processing DeleteTablet for tablet c1e736aea2b54b378ffbe821cb9e42c4 with delete_type TABLET_DATA_TOMBSTONED (TS 96cd069722b34c1eab8ae9a2f953dd7a not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:60834
I20251026 08:52:50.938767 4422 tablet_replica.cc:333] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:52:50.938921 4422 raft_consensus.cc:2243] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:50.939193 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c1e736aea2b54b378ffbe821cb9e42c4 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:50.939240 4422 raft_consensus.cc:2272] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:50.940651 4422 ts_tablet_manager.cc:1916] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:50.941746 3832 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.945564 3656 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.946220 3910 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.947213 4422 ts_tablet_manager.cc:1929] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20251026 08:52:50.947345 4422 log.cc:1199] T c1e736aea2b54b378ffbe821cb9e42c4 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/wals/c1e736aea2b54b378ffbe821cb9e42c4
I20251026 08:52:50.948041 3645 catalog_manager.cc:4985] TS 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): tablet c1e736aea2b54b378ffbe821cb9e42c4 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:50.953581 4225 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.956630 3760 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:50.958043 4225 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251026 08:52:50.958316 3832 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251026 08:52:50.958544 4376 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.958791 4411 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:50.960610 4411 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.961030 4225 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.961678 3832 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.963757 3646 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0409b48759d34a6fa9c86d88b809e093 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:50.964735 3657 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:50.967460 3898 tablet_service.cc:1552] Processing DeleteTablet for tablet 0409b48759d34a6fa9c86d88b809e093 with delete_type TABLET_DATA_TOMBSTONED (TS 53f35e5629a94c8f81ad198ebc4df6ae not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:47574
I20251026 08:52:50.967787 4418 tablet_replica.cc:333] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:52:50.967963 4418 raft_consensus.cc:2243] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:50.968281 4418 raft_consensus.cc:2272] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:50.968906 4418 ts_tablet_manager.cc:1916] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:50.972728 4418 ts_tablet_manager.cc:1929] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20251026 08:52:50.972808 4418 log.cc:1199] T 0409b48759d34a6fa9c86d88b809e093 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/wals/0409b48759d34a6fa9c86d88b809e093
I20251026 08:52:50.973274 3645 catalog_manager.cc:4985] TS 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): tablet 0409b48759d34a6fa9c86d88b809e093 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:51.582558 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:51.643213 3758 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "49e9ac1b52f24abf8690af5a6924646c"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:52:51.643411 3758 raft_consensus.cc:606] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Received request to transfer leadership to TS 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:52:51.644377 3910 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "381d5756da8c4fa3ab5663d92bf76739"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:51.644496 3910 raft_consensus.cc:606] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Received request to transfer leadership to TS 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:52:51.645284 3832 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:51.645394 3832 raft_consensus.cc:606] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:51.646134 3758 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:52:51.646239 3758 raft_consensus.cc:606] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:52:51.646888 3832 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:51.647015 3832 raft_consensus.cc:606] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:51.647703 3758 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "b983c9a7c2924030a396232a4ad46572"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:52:51.647810 3758 raft_consensus.cc:606] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Received request to transfer leadership to TS b983c9a7c2924030a396232a4ad46572
I20251026 08:52:51.648135 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20251026 08:52:51.648362 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:51.660916 4404 raft_consensus.cc:993] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:51.661056 4435 raft_consensus.cc:1081] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:51.661633 3910 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:52:51.661746 3910 raft_consensus.cc:493] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:51.661834 3910 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.663520 3910 raft_consensus.cc:515] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:51.663965 3910 leader_election.cc:290] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:51.664239 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:51.664268 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:51.664388 3758 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.664412 3832 raft_consensus.cc:3055] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:51.664471 3832 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:51.664662 3832 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:51.664901 3832 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.666731 3758 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:51.667089 3876 leader_election.cc:304] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:51.667096 3832 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:51.667323 4395 raft_consensus.cc:2804] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:51.667433 4395 raft_consensus.cc:697] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:51.667647 4395 consensus_queue.cc:237] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:51.669538 3656 catalog_manager.cc:5649] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 1 to 2, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:51.750418 4404 raft_consensus.cc:993] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:51.750553 4404 raft_consensus.cc:1081] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:51.751013 3910 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:52:51.751139 3910 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:51.751243 3910 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.752939 3910 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:51.753476 3910 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:51.753717 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:51.753788 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:51.753857 3758 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.753875 3832 raft_consensus.cc:3055] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:51.753958 3832 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:51.754176 3832 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:51.754429 3832 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.756146 3758 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:51.756479 3876 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:51.756531 3832 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:51.756680 4395 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:51.756798 4395 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:51.757015 4395 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:51.759095 3656 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 1 to 2, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:51.851253 4423 raft_consensus.cc:993] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower 381d5756da8c4fa3ab5663d92bf76739 to start an election
I20251026 08:52:51.851380 4423 raft_consensus.cc:1081] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Signalling peer 381d5756da8c4fa3ab5663d92bf76739 to start an election
I20251026 08:52:51.851938 4151 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e"
dest_uuid: "381d5756da8c4fa3ab5663d92bf76739"
from {username='slave'} at 127.0.0.1:33734
I20251026 08:52:51.852070 4151 raft_consensus.cc:493] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:51.852123 4151 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:51.853672 4151 raft_consensus.cc:515] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:51.854257 4151 leader_election.cc:290] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [CANDIDATE]: Term 3 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:51.854538 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "381d5756da8c4fa3ab5663d92bf76739" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:51.854591 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0b2c4bd03bfd4efd8aa306d8557fad3e" candidate_uuid: "381d5756da8c4fa3ab5663d92bf76739" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:51.854668 3832 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:51.854740 3910 raft_consensus.cc:3055] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:52:51.854801 3910 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:51.855005 3910 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:51.855259 3910 raft_consensus.cc:3060] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:51.856293 3832 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381d5756da8c4fa3ab5663d92bf76739 in term 3.
I20251026 08:52:51.856566 4116 leader_election.cc:304] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381d5756da8c4fa3ab5663d92bf76739, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:51.856954 3910 raft_consensus.cc:2468] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381d5756da8c4fa3ab5663d92bf76739 in term 3.
I20251026 08:52:51.857034 4441 raft_consensus.cc:2804] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:52:51.857389 4441 raft_consensus.cc:697] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEADER]: Becoming Leader. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Running, Role: LEADER
I20251026 08:52:51.857652 4441 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:51.860144 3656 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: term changed from 2 to 3, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68). New cstate: current_term: 3 leader_uuid: "381d5756da8c4fa3ab5663d92bf76739" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:51.878119 4412 raft_consensus.cc:993] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower b983c9a7c2924030a396232a4ad46572 to start an election
I20251026 08:52:51.878278 4412 raft_consensus.cc:1081] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Signalling peer b983c9a7c2924030a396232a4ad46572 to start an election
I20251026 08:52:51.878741 4300 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5"
dest_uuid: "b983c9a7c2924030a396232a4ad46572"
from {username='slave'} at 127.0.0.1:52940
I20251026 08:52:51.878851 4300 raft_consensus.cc:493] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:51.878934 4300 raft_consensus.cc:3060] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.880584 4300 raft_consensus.cc:515] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:51.881101 4300 leader_election.cc:290] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:51.881454 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "b983c9a7c2924030a396232a4ad46572" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:51.881597 3758 raft_consensus.cc:3055] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:51.881668 3758 raft_consensus.cc:740] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:51.881870 3758 consensus_queue.cc:260] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:51.882206 3758 raft_consensus.cc:3060] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.883985 3758 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b983c9a7c2924030a396232a4ad46572 in term 2.
I20251026 08:52:51.884317 4266 leader_election.cc:304] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b983c9a7c2924030a396232a4ad46572, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:51.884846 4446 raft_consensus.cc:2804] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:51.885214 4446 raft_consensus.cc:697] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEADER]: Becoming Leader. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Running, Role: LEADER
I20251026 08:52:51.885430 4446 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:51.886590 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c5f200f20b1741b4a2c6bc379b2ae8f5" candidate_uuid: "b983c9a7c2924030a396232a4ad46572" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:51.886703 3832 raft_consensus.cc:3060] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.887310 3656 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 reported cstate change: term changed from 1 to 2, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to b983c9a7c2924030a396232a4ad46572 (127.0.103.70). New cstate: current_term: 2 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:51.888567 3832 raft_consensus.cc:2468] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b983c9a7c2924030a396232a4ad46572 in term 2.
I20251026 08:52:51.908339 4417 raft_consensus.cc:993] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:51.908461 4417 raft_consensus.cc:1081] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:51.908929 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60858
I20251026 08:52:51.909067 3835 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:51.909139 3835 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.910735 3835 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:51.911136 3835 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:51.911473 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:51.911520 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:51.911602 3758 raft_consensus.cc:3055] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:51.911679 3758 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:51.911691 3910 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.911868 3758 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:51.912082 3758 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:51.913486 3910 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 2.
I20251026 08:52:51.913739 3758 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 2.
I20251026 08:52:51.913838 3801 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:51.914041 4404 raft_consensus.cc:2804] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:51.914206 4404 raft_consensus.cc:697] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:51.914506 4404 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:51.916496 3656 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 1 to 2, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 2 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:51.969465 3758 consensus_queue.cc:237] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:51.971539 3835 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20251026 08:52:51.971843 3726 consensus_peers.cc:597] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c430afe1db8423b97c3c436cb912760. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:51.972235 3910 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:51.972218 4417 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:51.973670 4417 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:51.975356 4412 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:51.975827 3835 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:51.979987 3656 catalog_manager.cc:5649] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:51.981343 3907 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:51.986014 3907 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:51.988049 3758 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
W20251026 08:52:51.988095 3873 consensus_peers.cc:597] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f3c583b37d74efe9630e3723a2c313e. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:51.988432 3835 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 1 index: 27. (index mismatch)
I20251026 08:52:51.988633 4456 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:51.989483 4456 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:51.991008 4395 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:51.992444 3835 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:51.993799 3758 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.27: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
W20251026 08:52:51.996043 3707 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20251026 08:52:51.996863 3656 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index -1 to 27, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
W20251026 08:52:52.000952 3707 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 9ee117517b8b4ae18fa61970868914a4, TS 96cd069722b34c1eab8ae9a2f953dd7a -> TS 381d5756da8c4fa3ab5663d92bf76739 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
W20251026 08:52:52.001010 3707 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 9ee117517b8b4ae18fa61970868914a4, TS 96cd069722b34c1eab8ae9a2f953dd7a -> TS 381d5756da8c4fa3ab5663d92bf76739 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
W20251026 08:52:52.004237 3707 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b1edddc92bb0463b9ed275da3c7154ef, TS 53f35e5629a94c8f81ad198ebc4df6ae -> TS b983c9a7c2924030a396232a4ad46572 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
W20251026 08:52:52.004297 3707 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b1edddc92bb0463b9ed275da3c7154ef, TS 53f35e5629a94c8f81ad198ebc4df6ae -> TS b983c9a7c2924030a396232a4ad46572 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
W20251026 08:52:52.007256 3707 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 3d3135b8b6fe4dff81329c307264ea9f, TS e7be295d3ba247a1a4475fa020d1ade1 -> TS 49e9ac1b52f24abf8690af5a6924646c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
W20251026 08:52:52.007316 3707 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 3d3135b8b6fe4dff81329c307264ea9f, TS e7be295d3ba247a1a4475fa020d1ade1 -> TS 49e9ac1b52f24abf8690af5a6924646c move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
W20251026 08:52:52.015115 3707 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c5f200f20b1741b4a2c6bc379b2ae8f5, TS 96cd069722b34c1eab8ae9a2f953dd7a -> TS 381d5756da8c4fa3ab5663d92bf76739 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
W20251026 08:52:52.015213 3707 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c5f200f20b1741b4a2c6bc379b2ae8f5, TS 96cd069722b34c1eab8ae9a2f953dd7a -> TS 381d5756da8c4fa3ab5663d92bf76739 move failed, destination replica disappeared from tablet's Raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:52.090590 3835 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:52.091205 4395 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:52.095888 3760 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:52.097523 4395 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:52.129034 4412 raft_consensus.cc:993] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 49e9ac1b52f24abf8690af5a6924646c to start an election
I20251026 08:52:52.129159 4412 raft_consensus.cc:1081] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Signalling peer 49e9ac1b52f24abf8690af5a6924646c to start an election
I20251026 08:52:52.129669 4225 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093"
dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
from {username='slave'} at 127.0.0.1:47568
I20251026 08:52:52.129810 4225 raft_consensus.cc:493] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:52.129910 4225 raft_consensus.cc:3060] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:52.132426 4225 raft_consensus.cc:515] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:52.133244 4225 leader_election.cc:290] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:52.133562 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "49e9ac1b52f24abf8690af5a6924646c" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:52.133715 3758 raft_consensus.cc:3055] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:52.133795 3758 raft_consensus.cc:740] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:52.134047 3758 consensus_queue.cc:260] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:52.134449 3758 raft_consensus.cc:3060] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:52.136585 3758 raft_consensus.cc:2468] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e9ac1b52f24abf8690af5a6924646c in term 2.
I20251026 08:52:52.136955 4192 leader_election.cc:304] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9ac1b52f24abf8690af5a6924646c, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:52.137704 4467 raft_consensus.cc:2804] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:52.138248 4467 raft_consensus.cc:697] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEADER]: Becoming Leader. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Running, Role: LEADER
I20251026 08:52:52.138537 4467 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:52.140774 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0409b48759d34a6fa9c86d88b809e093" candidate_uuid: "49e9ac1b52f24abf8690af5a6924646c" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:52.140895 3835 raft_consensus.cc:3060] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:52.142771 3835 raft_consensus.cc:2468] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e9ac1b52f24abf8690af5a6924646c in term 2.
I20251026 08:52:52.143424 3657 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: term changed from 1 to 2, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69). New cstate: current_term: 2 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:52.230957 3760 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:52.231454 4395 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:52.235575 3835 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:52.236011 4395 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:52.256628 3907 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251026 08:52:52.256997 4441 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:52.260455 3835 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251026 08:52:52.260914 4441 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:52.316602 3760 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:52.317081 4446 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:52.328413 3835 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:52.329092 4446 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:52.404665 4479 ts_tablet_manager.cc:933] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: Initiating tablet copy from peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:52.405117 4479 tablet_copy_client.cc:323] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:39515
I20251026 08:52:52.405630 3771 tablet_copy_service.cc:140] P e7be295d3ba247a1a4475fa020d1ade1: Received BeginTabletCopySession request for tablet 8c430afe1db8423b97c3c436cb912760 from peer 49e9ac1b52f24abf8690af5a6924646c ({username='slave'} at 127.0.0.1:46316)
I20251026 08:52:52.405741 3771 tablet_copy_service.cc:161] P e7be295d3ba247a1a4475fa020d1ade1: Beginning new tablet copy session on tablet 8c430afe1db8423b97c3c436cb912760 from peer 49e9ac1b52f24abf8690af5a6924646c at {username='slave'} at 127.0.0.1:46316: session id = 49e9ac1b52f24abf8690af5a6924646c-8c430afe1db8423b97c3c436cb912760
I20251026 08:52:52.408234 3771 tablet_copy_source_session.cc:215] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:52.409205 4479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c430afe1db8423b97c3c436cb912760. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:52.414078 4479 tablet_copy_client.cc:806] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:52.414357 4479 tablet_copy_client.cc:670] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:52.416162 4480 tablet_copy_client.cc:1025] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:52.432653 4481 ts_tablet_manager.cc:933] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:52.433369 4481 tablet_copy_client.cc:323] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:52.433944 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet 5f3c583b37d74efe9630e3723a2c313e from peer b983c9a7c2924030a396232a4ad46572 ({username='slave'} at 127.0.0.1:47646)
I20251026 08:52:52.434089 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet 5f3c583b37d74efe9630e3723a2c313e from peer b983c9a7c2924030a396232a4ad46572 at {username='slave'} at 127.0.0.1:47646: session id = b983c9a7c2924030a396232a4ad46572-5f3c583b37d74efe9630e3723a2c313e
I20251026 08:52:52.436823 3920 tablet_copy_source_session.cc:215] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:52.437816 4481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3c583b37d74efe9630e3723a2c313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:52.442940 4481 tablet_copy_client.cc:806] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:52.443164 4481 tablet_copy_client.cc:670] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:52.444420 4482 tablet_copy_client.cc:1025] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:52.445654 3910 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:52.446027 4435 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:52.449012 3758 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:52.449760 4404 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:52.579278 3758 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:52.579872 4467 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:52.584565 3835 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:52.585019 4488 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:53.648710 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:53.651707 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:53.651841 3907 raft_consensus.cc:606] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:53.652709 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:53.652822 3907 raft_consensus.cc:606] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:53.653611 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:53.653712 3835 raft_consensus.cc:606] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:53.654050 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251026 08:52:53.654363 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:53.750923 4501 raft_consensus.cc:993] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:53.751091 4501 raft_consensus.cc:1081] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:53.751610 3907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:52:53.751744 3907 raft_consensus.cc:493] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:53.751833 3907 raft_consensus.cc:3060] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:53.753899 3907 raft_consensus.cc:515] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:53.754700 3907 leader_election.cc:290] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881)
I20251026 08:52:53.754820 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:53.754882 4151 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d0a49b69e0a94ae9827e91c43006c2d5" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: true dest_uuid: "381d5756da8c4fa3ab5663d92bf76739"
I20251026 08:52:53.754967 3835 raft_consensus.cc:3055] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:53.755055 3835 raft_consensus.cc:740] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:53.755077 4151 raft_consensus.cc:3060] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:53.755286 3835 consensus_queue.cc:260] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:53.755508 3835 raft_consensus.cc:3060] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:53.756778 4151 raft_consensus.cc:2468] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:53.757189 3835 raft_consensus.cc:2468] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:53.757120 3876 leader_election.cc:304] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381d5756da8c4fa3ab5663d92bf76739, 53f35e5629a94c8f81ad198ebc4df6ae; no voters:
I20251026 08:52:53.757439 4500 raft_consensus.cc:2804] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:53.757547 4500 raft_consensus.cc:697] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:53.757761 4500 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:53.760360 3653 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 1 to 2, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:53.938439 4499 raft_consensus.cc:993] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:53.938611 4499 raft_consensus.cc:1081] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:53.939262 3758 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51954
I20251026 08:52:53.939460 3758 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:53.939584 3758 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:53.941623 3758 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:53.942633 3758 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 3 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:53.942788 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:53.942952 3832 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:53.944430 3832 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 3.
I20251026 08:52:53.944815 3726 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:53.944978 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:53.945060 4498 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:52:53.945122 3907 raft_consensus.cc:3055] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:52:53.945168 3907 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:53.945206 4498 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:53.945348 3907 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:53.945469 4498 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:53.945645 3907 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:53.947961 3907 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 3.
I20251026 08:52:53.948262 3653 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 2 to 3, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:54.019958 4500 raft_consensus.cc:993] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:54.020088 4500 raft_consensus.cc:1081] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:54.020645 3760 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51954
I20251026 08:52:54.020839 3760 raft_consensus.cc:493] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:54.020931 3760 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:54.023239 3760 raft_consensus.cc:515] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.023834 3760 leader_election.cc:290] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 2 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:54.024165 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:54.024320 3832 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:54.024322 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:54.025979 3907 raft_consensus.cc:3055] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:54.026087 3907 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:54.026387 3907 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.026677 3907 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:54.026952 3832 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 2.
I20251026 08:52:54.027290 3726 leader_election.cc:304] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:54.027532 4498 raft_consensus.cc:2804] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:54.027642 4498 raft_consensus.cc:697] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:54.027864 4498 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.028625 3907 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 2.
I20251026 08:52:54.030385 3653 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 1 to 2, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:54.034611 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:54.034719 3760 raft_consensus.cc:606] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership
I20251026 08:52:54.037341 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:54.037453 3760 raft_consensus.cc:606] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership
W20251026 08:52:54.037788 3646 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 5f3c583b37d74efe9630e3723a2c313e already in progress [suppressed 2 similar messages]
I20251026 08:52:54.038990 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:54.039157 3760 raft_consensus.cc:606] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership
W20251026 08:52:54.039564 3707 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 5f3c583b37d74efe9630e3723a2c313e already in progress
W20251026 08:52:54.039621 3707 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 5f3c583b37d74efe9630e3723a2c313e already in progress
I20251026 08:52:54.272207 4151 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:54.272776 4500 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:54.277760 3832 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:54.278299 4500 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:54.412894 3910 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20251026 08:52:54.413280 4498 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:54.416723 4479 tablet_copy_client.cc:538] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:54.419008 3832 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20251026 08:52:54.419550 4518 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:54.420552 4479 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap starting.
I20251026 08:52:54.444748 4481 tablet_copy_client.cc:538] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:54.447391 4481 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: Bootstrap starting.
I20251026 08:52:54.449558 4479 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=108 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:54.449872 4479 tablet_bootstrap.cc:492] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap complete.
I20251026 08:52:54.450057 4479 ts_tablet_manager.cc:1403] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: Time spent bootstrapping tablet: real 0.030s user 0.025s sys 0.005s
I20251026 08:52:54.450788 4479 raft_consensus.cc:359] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:54.450945 4479 raft_consensus.cc:740] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Initialized, Role: LEARNER
I20251026 08:52:54.451112 4479 consensus_queue.cc:260] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:54.451617 4479 ts_tablet_manager.cc:1434] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:54.452395 3771 tablet_copy_service.cc:342] P e7be295d3ba247a1a4475fa020d1ade1: Request end of tablet copy session 49e9ac1b52f24abf8690af5a6924646c-8c430afe1db8423b97c3c436cb912760 received from {username='slave'} at 127.0.0.1:46316
I20251026 08:52:54.452478 3771 tablet_copy_service.cc:434] P e7be295d3ba247a1a4475fa020d1ade1: ending tablet copy session 49e9ac1b52f24abf8690af5a6924646c-8c430afe1db8423b97c3c436cb912760 on tablet 8c430afe1db8423b97c3c436cb912760 with peer 49e9ac1b52f24abf8690af5a6924646c
W20251026 08:52:54.472237 3724 consensus_peers.cc:597] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:54.480782 4481 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: Bootstrap replayed 1/1 log segments. Stats: ops{read=27 overwritten=0 applied=27 ignored=0} inserts{seen=86 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:54.481107 4481 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: Bootstrap complete.
I20251026 08:52:54.481349 4481 ts_tablet_manager.cc:1403] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.004s
I20251026 08:52:54.481868 4481 raft_consensus.cc:359] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.482107 4481 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Initialized, Role: LEARNER
I20251026 08:52:54.482336 4481 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 27, Last appended: 1.27, Last appended by leader: 27, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.482788 4481 ts_tablet_manager.cc:1434] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 08:52:54.483354 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session b983c9a7c2924030a396232a4ad46572-5f3c583b37d74efe9630e3723a2c313e received from {username='slave'} at 127.0.0.1:47646
I20251026 08:52:54.483449 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session b983c9a7c2924030a396232a4ad46572-5f3c583b37d74efe9630e3723a2c313e on tablet 5f3c583b37d74efe9630e3723a2c313e with peer b983c9a7c2924030a396232a4ad46572
I20251026 08:52:54.525131 3832 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:54.525601 4518 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:54.528798 4498 raft_consensus.cc:993] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:54.528892 4498 raft_consensus.cc:1081] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:54.529748 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60858
I20251026 08:52:54.529897 3835 raft_consensus.cc:493] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:54.529966 3835 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:54.530085 3910 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:54.530529 4498 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:54.532553 3835 raft_consensus.cc:515] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.533121 3835 leader_election.cc:290] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 3 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:54.533392 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:54.533532 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5f3c583b37d74efe9630e3723a2c313e" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:54.533542 3760 raft_consensus.cc:3055] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:52:54.533679 3760 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:54.533679 3910 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:54.533895 3760 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.534291 3760 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:54.535882 3760 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 3.
I20251026 08:52:54.536203 3802 leader_election.cc:304] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:54.536203 3910 raft_consensus.cc:2468] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 3.
I20251026 08:52:54.536648 4496 raft_consensus.cc:2804] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:52:54.537827 4496 raft_consensus.cc:697] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:54.538105 4496 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:54.540375 3657 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 2 to 3, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 27 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:54.785470 4225 raft_consensus.cc:1217] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.26->[1.27-1.27] Dedup: 1.27->[]
I20251026 08:52:54.927753 4300 raft_consensus.cc:3060] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 1 LEARNER]: Advancing to term 3
I20251026 08:52:54.929847 4300 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 LEARNER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20251026 08:52:54.930272 4496 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:54.932780 4524 raft_consensus.cc:1064] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
W20251026 08:52:54.932895 4524 raft_consensus.cc:1068] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: Unable to promote non-voter b983c9a7c2924030a396232a4ad46572: Illegal state: Leader has not yet committed an operation in its own term
I20251026 08:52:55.134266 3758 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20251026 08:52:55.134810 4524 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:55.135133 3907 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 3 index: 29. (index mismatch)
I20251026 08:52:55.135617 4524 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:55.145483 4496 raft_consensus.cc:1064] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
I20251026 08:52:55.146157 4496 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 3.29, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:55.147863 4300 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 LEARNER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251026 08:52:55.148196 3907 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251026 08:52:55.148347 4524 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:55.148871 4524 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:55.151147 4534 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Committing config change with OpId 3.30: config changed from index 27 to 30, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.154511 3907 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.157181 4300 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.158890 3758 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 29. Preceding OpId from leader: term: 3 index: 30. (index mismatch)
I20251026 08:52:55.160470 4538 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:55.161646 3653 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 27 to 30, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.162412 3758 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Committing config change with OpId 3.30: config changed from index 27 to 30, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.170882 3835 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:55.172237 4300 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:52:55.172518 3907 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:52:55.172983 4496 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:52:55.173312 4496 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:52:55.174921 4496 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.176173 3907 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.177778 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 5f3c583b37d74efe9630e3723a2c313e with cas_config_opid_index 30: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:55.177927 4300 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 30 to 31, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:55.178879 3650 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 30 to 31, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:55.182296 3748 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f3c583b37d74efe9630e3723a2c313e with delete_type TABLET_DATA_TOMBSTONED (TS e7be295d3ba247a1a4475fa020d1ade1 not found in new config with opid_index 31) from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:55.182906 4543 tablet_replica.cc:333] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:52:55.183095 4543 raft_consensus.cc:2243] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:55.183463 4543 raft_consensus.cc:2272] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:55.184194 4543 ts_tablet_manager.cc:1916] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:55.188632 4543 ts_tablet_manager.cc:1929] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.30
I20251026 08:52:55.188704 4543 log.cc:1199] T 5f3c583b37d74efe9630e3723a2c313e P e7be295d3ba247a1a4475fa020d1ade1: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/wals/5f3c583b37d74efe9630e3723a2c313e
I20251026 08:52:55.189364 3646 catalog_manager.cc:4985] TS e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): tablet 5f3c583b37d74efe9630e3723a2c313e (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:55.224351 4498 raft_consensus.cc:1064] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: attempting to promote NON_VOTER 49e9ac1b52f24abf8690af5a6924646c to VOTER
I20251026 08:52:55.224946 4498 consensus_queue.cc:237] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:55.226742 3907 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:55.226797 4225 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 LEARNER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:55.226797 3835 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20251026 08:52:55.227375 4497 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:55.227646 4518 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:55.228318 4497 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:55.231606 3835 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.234257 4518 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.238626 3907 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.239547 3656 catalog_manager.cc:5649] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:55.244024 4225 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index 27 to 28, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.246403 3760 consensus_queue.cc:237] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 1.28, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:55.247714 3835 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251026 08:52:55.248028 4225 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 28. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20251026 08:52:55.248672 4518 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:55.248875 4498 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:55.250651 4545 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.251276 3835 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.251389 4225 raft_consensus.cc:2955] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Committing config change with OpId 1.29: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:55.254607 3646 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 8c430afe1db8423b97c3c436cb912760 with cas_config_opid_index 28: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:55.254814 3657 catalog_manager.cc:5649] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 28 to 29, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New cstate: current_term: 1 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:52:55.257633 3898 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c430afe1db8423b97c3c436cb912760 with delete_type TABLET_DATA_TOMBSTONED (TS 53f35e5629a94c8f81ad198ebc4df6ae not found in new config with opid_index 29) from {username='slave'} at 127.0.0.1:47574
I20251026 08:52:55.258311 4549 tablet_replica.cc:333] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:52:55.258476 4549 raft_consensus.cc:2243] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:55.258697 4549 raft_consensus.cc:2272] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:55.259389 4549 ts_tablet_manager.cc:1916] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:55.262900 4549 ts_tablet_manager.cc:1929] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.28
I20251026 08:52:55.262974 4549 log.cc:1199] T 8c430afe1db8423b97c3c436cb912760 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/wals/8c430afe1db8423b97c3c436cb912760
I20251026 08:52:55.263458 3645 catalog_manager.cc:4985] TS 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): tablet 8c430afe1db8423b97c3c436cb912760 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:55.654722 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:55.657403 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:55.657611 3907 raft_consensus.cc:606] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:52:55.658624 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:52:55.658759 3760 raft_consensus.cc:606] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:52:55.659667 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:55.659823 3835 raft_consensus.cc:606] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:55.660194 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 3
I20251026 08:52:55.660418 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:55.788825 4496 raft_consensus.cc:993] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:55.788960 4496 raft_consensus.cc:1081] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:55.789433 3907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:52:55.789580 3907 raft_consensus.cc:493] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:55.789641 3907 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:55.791337 3907 raft_consensus.cc:515] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:55.791759 3907 leader_election.cc:290] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:55.792104 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:55.792152 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:55.792266 3760 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:55.792281 3835 raft_consensus.cc:3055] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:55.792352 3835 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:55.792550 3835 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:55.792811 3835 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:55.794569 3760 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:55.794888 3876 leader_election.cc:304] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:55.794929 3835 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 2.
I20251026 08:52:55.795084 4500 raft_consensus.cc:2804] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:55.795599 4500 raft_consensus.cc:697] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:55.795818 4500 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 1.26, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:55.797847 3656 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 1 to 2, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:56.054715 4545 raft_consensus.cc:993] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:56.054862 4545 raft_consensus.cc:1081] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:56.055426 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60858
I20251026 08:52:56.055603 3835 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:56.055683 3835 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:52:56.057890 3835 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:56.058404 3835 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 4 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:56.058844 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:56.058844 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:56.059036 3907 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:52:56.059055 3760 raft_consensus.cc:3055] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 08:52:56.059139 3760 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:56.059384 3760 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:56.059674 3760 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:52:56.061632 3907 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 4.
I20251026 08:52:56.062016 3760 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 4.
I20251026 08:52:56.062040 3801 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:56.062287 4496 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Leader election won for term 4
I20251026 08:52:56.062388 4496 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:56.062590 4496 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:52:56.064843 3656 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 3 to 4, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 4 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:56.202036 3760 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:56.202507 4500 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:56.205413 3835 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 26. Preceding OpId from leader: term: 2 index: 27. (index mismatch)
I20251026 08:52:56.205873 4551 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20251026 08:52:56.208992 4560 raft_consensus.cc:993] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:56.209124 4551 raft_consensus.cc:1081] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:52:56.210296 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60842
I20251026 08:52:56.210427 3835 raft_consensus.cc:493] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:56.210515 3835 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:56.212476 3835 raft_consensus.cc:515] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:56.212954 3835 leader_election.cc:290] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 3 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:52:56.213274 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:56.213361 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:56.213423 3907 raft_consensus.cc:3055] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:52:56.213466 3760 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:56.213486 3907 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:56.213719 3907 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:56.213953 3907 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:56.215939 3760 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 3.
I20251026 08:52:56.216269 3907 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 3.
I20251026 08:52:56.216275 3802 leader_election.cc:304] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:56.216518 4496 raft_consensus.cc:2804] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:52:56.216639 4496 raft_consensus.cc:697] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:56.216845 4496 consensus_queue.cc:237] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 27, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:56.218724 3656 catalog_manager.cc:5649] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 2 to 3, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } }
I20251026 08:52:56.263993 4225 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:56.265959 3835 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:52:56.266170 3760 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:52:56.266532 4568 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:52:56.266785 4467 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:52:56.269690 4467 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.270267 3835 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.271064 3760 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 29 to 31, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
W20251026 08:52:56.273926 4192 consensus_peers.cc:597] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0409b48759d34a6fa9c86d88b809e093. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:56.274562 3832 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:56.274528 3650 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: config changed from index 29 to 31, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:56.276458 3907 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:52:56.276551 4300 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
W20251026 08:52:56.277065 3802 consensus_peers.cc:597] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f3c583b37d74efe9630e3723a2c313e. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:56.277139 4496 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:52:56.278775 4550 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:52:56.282691 4496 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.283257 3907 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.286993 4300 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.288177 3657 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 31 to 32, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.291635 3907 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:56.293160 3760 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:56.293395 3832 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
W20251026 08:52:56.293537 3875 consensus_peers.cc:597] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1edddc92bb0463b9ed275da3c7154ef. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:56.293567 4551 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:56.293892 4551 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:56.295081 4551 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:56.295845 3832 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:56.298357 3760 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:52:56.299306 3650 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index -1 to 28, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:56.300050 3907 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:56.302747 4151 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:56.302867 3832 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:56.303215 4500 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
W20251026 08:52:56.303473 3873 consensus_peers.cc:597] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d0a49b69e0a94ae9827e91c43006c2d5. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:56.304637 4551 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:56.307060 4500 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.310978 3835 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.312143 4151 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.312515 3907 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:56.315212 3657 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 28 to 30, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.317859 4300 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
I20251026 08:52:56.317914 3760 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 1 index: 30. (index mismatch)
W20251026 08:52:56.318182 3876 consensus_peers.cc:597] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c1e736aea2b54b378ffbe821cb9e42c4. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:56.318382 4500 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:56.319056 4581 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:56.321187 4581 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.322583 4300 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.322708 3760 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Committing config change with OpId 1.30: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.327067 3656 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 reported cstate change: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:52:56.327486 3835 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 2.27, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:56.329665 3907 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:56.330250 4574 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:56.330318 3760 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 2 index: 28. (index mismatch)
I20251026 08:52:56.330713 4577 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
W20251026 08:52:56.330869 3799 consensus_peers.cc:597] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dd57692a7704c9690a892319e4bf446. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:56.332945 4550 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.333606 3760 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.333629 3907 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Committing config change with OpId 2.28: config changed from index -1 to 28, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:52:56.336666 3656 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index -1 to 28, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New cstate: current_term: 2 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:56.545537 3760 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20251026 08:52:56.546043 4574 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:56.561962 3907 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20251026 08:52:56.562703 4550 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:56.661104 4590 ts_tablet_manager.cc:933] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: Initiating tablet copy from peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:52:56.661764 4590 tablet_copy_client.cc:323] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.69:34723
I20251026 08:52:56.666899 4236 tablet_copy_service.cc:140] P 49e9ac1b52f24abf8690af5a6924646c: Received BeginTabletCopySession request for tablet 0409b48759d34a6fa9c86d88b809e093 from peer 381d5756da8c4fa3ab5663d92bf76739 ({username='slave'} at 127.0.0.1:58254)
I20251026 08:52:56.667049 4236 tablet_copy_service.cc:161] P 49e9ac1b52f24abf8690af5a6924646c: Beginning new tablet copy session on tablet 0409b48759d34a6fa9c86d88b809e093 from peer 381d5756da8c4fa3ab5663d92bf76739 at {username='slave'} at 127.0.0.1:58254: session id = 381d5756da8c4fa3ab5663d92bf76739-0409b48759d34a6fa9c86d88b809e093
I20251026 08:52:56.669142 4236 tablet_copy_source_session.cc:215] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:56.670010 4590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0409b48759d34a6fa9c86d88b809e093. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:56.674106 4590 tablet_copy_client.cc:806] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:56.674314 4590 tablet_copy_client.cc:670] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:56.675568 4591 tablet_copy_client.cc:1025] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:56.692380 4593 ts_tablet_manager.cc:933] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:56.693050 4593 tablet_copy_client.cc:323] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:56.693634 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet b1edddc92bb0463b9ed275da3c7154ef from peer 49e9ac1b52f24abf8690af5a6924646c ({username='slave'} at 127.0.0.1:47636)
I20251026 08:52:56.693749 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet b1edddc92bb0463b9ed275da3c7154ef from peer 49e9ac1b52f24abf8690af5a6924646c at {username='slave'} at 127.0.0.1:47636: session id = 49e9ac1b52f24abf8690af5a6924646c-b1edddc92bb0463b9ed275da3c7154ef
I20251026 08:52:56.695786 3920 tablet_copy_source_session.cc:215] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:56.696759 4593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1edddc92bb0463b9ed275da3c7154ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:56.698681 3760 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20251026 08:52:56.699162 4550 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:56.704263 4593 tablet_copy_client.cc:806] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:56.704701 4593 tablet_copy_client.cc:670] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:56.707158 4594 tablet_copy_client.cc:1025] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:56.708688 3910 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 27. Preceding OpId from leader: term: 3 index: 28. (index mismatch)
I20251026 08:52:56.709113 4577 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20251026 08:52:56.779587 4597 ts_tablet_manager.cc:933] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: Initiating tablet copy from peer 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:56.780180 4597 tablet_copy_client.cc:323] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:37913
I20251026 08:52:56.780840 3846 tablet_copy_service.cc:140] P 96cd069722b34c1eab8ae9a2f953dd7a: Received BeginTabletCopySession request for tablet 5f3c583b37d74efe9630e3723a2c313e from peer 381d5756da8c4fa3ab5663d92bf76739 ({username='slave'} at 127.0.0.1:42606)
I20251026 08:52:56.780956 3846 tablet_copy_service.cc:161] P 96cd069722b34c1eab8ae9a2f953dd7a: Beginning new tablet copy session on tablet 5f3c583b37d74efe9630e3723a2c313e from peer 381d5756da8c4fa3ab5663d92bf76739 at {username='slave'} at 127.0.0.1:42606: session id = 381d5756da8c4fa3ab5663d92bf76739-5f3c583b37d74efe9630e3723a2c313e
I20251026 08:52:56.784426 3846 tablet_copy_source_session.cc:215] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:56.785661 4597 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f3c583b37d74efe9630e3723a2c313e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:56.790446 4599 ts_tablet_manager.cc:933] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: Initiating tablet copy from peer 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:56.791034 4599 tablet_copy_client.cc:323] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:37913
I20251026 08:52:56.791515 3846 tablet_copy_service.cc:140] P 96cd069722b34c1eab8ae9a2f953dd7a: Received BeginTabletCopySession request for tablet 6dd57692a7704c9690a892319e4bf446 from peer b983c9a7c2924030a396232a4ad46572 ({username='slave'} at 127.0.0.1:42620)
I20251026 08:52:56.791545 4597 tablet_copy_client.cc:806] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:56.791692 3846 tablet_copy_service.cc:161] P 96cd069722b34c1eab8ae9a2f953dd7a: Beginning new tablet copy session on tablet 6dd57692a7704c9690a892319e4bf446 from peer b983c9a7c2924030a396232a4ad46572 at {username='slave'} at 127.0.0.1:42620: session id = b983c9a7c2924030a396232a4ad46572-6dd57692a7704c9690a892319e4bf446
I20251026 08:52:56.791823 4597 tablet_copy_client.cc:670] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:56.793380 4598 tablet_copy_client.cc:1025] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:56.794164 3846 tablet_copy_source_session.cc:215] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:56.795058 4599 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd57692a7704c9690a892319e4bf446. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:56.800753 4599 tablet_copy_client.cc:806] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:56.800964 4599 tablet_copy_client.cc:670] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:56.802280 4600 tablet_copy_client.cc:1025] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:56.887606 4601 ts_tablet_manager.cc:933] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:56.888173 4601 tablet_copy_client.cc:323] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:56.888756 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet c1e736aea2b54b378ffbe821cb9e42c4 from peer 381d5756da8c4fa3ab5663d92bf76739 ({username='slave'} at 127.0.0.1:47624)
I20251026 08:52:56.888875 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet c1e736aea2b54b378ffbe821cb9e42c4 from peer 381d5756da8c4fa3ab5663d92bf76739 at {username='slave'} at 127.0.0.1:47624: session id = 381d5756da8c4fa3ab5663d92bf76739-c1e736aea2b54b378ffbe821cb9e42c4
I20251026 08:52:56.890676 3920 tablet_copy_source_session.cc:215] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:56.891620 4601 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e736aea2b54b378ffbe821cb9e42c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:56.895686 4601 tablet_copy_client.cc:806] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:56.895884 4601 tablet_copy_client.cc:670] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:56.897018 4602 tablet_copy_client.cc:1025] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:56.913130 4603 ts_tablet_manager.cc:933] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:56.913646 4603 tablet_copy_client.cc:323] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:52:56.914124 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet d0a49b69e0a94ae9827e91c43006c2d5 from peer b983c9a7c2924030a396232a4ad46572 ({username='slave'} at 127.0.0.1:47646)
I20251026 08:52:56.914227 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet d0a49b69e0a94ae9827e91c43006c2d5 from peer b983c9a7c2924030a396232a4ad46572 at {username='slave'} at 127.0.0.1:47646: session id = b983c9a7c2924030a396232a4ad46572-d0a49b69e0a94ae9827e91c43006c2d5
I20251026 08:52:56.916159 3920 tablet_copy_source_session.cc:215] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:52:56.917052 4603 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a49b69e0a94ae9827e91c43006c2d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:52:56.921763 4603 tablet_copy_client.cc:806] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 0 data blocks...
I20251026 08:52:56.921976 4603 tablet_copy_client.cc:670] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 1 WAL segments...
I20251026 08:52:56.923056 4604 tablet_copy_client.cc:1025] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Injecting latency into file download: 2000
I20251026 08:52:57.660780 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:57.664055 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:57.664213 3835 raft_consensus.cc:606] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:57.665082 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:57.665223 3907 raft_consensus.cc:606] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:57.665577 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 08:52:57.665748 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:57.707792 4550 raft_consensus.cc:993] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:57.707942 4550 raft_consensus.cc:1081] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:52:57.708515 3907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:52:57.708722 3907 raft_consensus.cc:493] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:57.708837 3907 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:52:57.711284 3907 raft_consensus.cc:515] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:57.712102 3907 leader_election.cc:290] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 4 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:52:57.712147 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:52:57.712335 3835 raft_consensus.cc:3055] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Stepping down as leader of term 3
I20251026 08:52:57.712416 3835 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:52:57.712647 3835 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:57.712898 3835 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:52:57.715015 3835 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 4.
I20251026 08:52:57.715282 3760 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:57.715350 3875 leader_election.cc:304] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:52:57.715404 3760 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:52:57.715616 4607 raft_consensus.cc:2804] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Leader election won for term 4
I20251026 08:52:57.715729 4607 raft_consensus.cc:697] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:57.715905 4607 consensus_queue.cc:237] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 3.28, Last appended by leader: 28, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:52:57.717700 3760 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 4.
I20251026 08:52:57.719354 3650 catalog_manager.cc:5649] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 3 to 4, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 4 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:57.739923 4606 raft_consensus.cc:993] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:57.740070 4606 raft_consensus.cc:1081] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:52:57.740643 3760 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51954
I20251026 08:52:57.740788 3760 raft_consensus.cc:493] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:57.740875 3760 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:57.743158 3760 raft_consensus.cc:515] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:57.743717 3760 leader_election.cc:290] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 2 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:52:57.744063 4300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "b983c9a7c2924030a396232a4ad46572"
I20251026 08:52:57.744206 4300 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:57.746145 4300 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 2.
I20251026 08:52:57.746219 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:57.746351 3907 raft_consensus.cc:3055] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:52:57.746418 3907 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:52:57.746496 3724 leader_election.cc:304] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b983c9a7c2924030a396232a4ad46572, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:52:57.746681 4545 raft_consensus.cc:2804] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:52:57.746985 4545 raft_consensus.cc:697] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:57.747207 4545 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:57.749740 3650 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 1 to 2, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 2 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:57.761412 3907 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:57.761646 3907 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:52:57.763083 3907 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 2.
I20251026 08:52:57.769639 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:57.769748 3760 raft_consensus.cc:606] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership
I20251026 08:52:57.776142 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:57.776252 3760 raft_consensus.cc:606] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership
I20251026 08:52:57.777233 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:57.777340 3760 raft_consensus.cc:606] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership
W20251026 08:52:57.777765 3707 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for c1e736aea2b54b378ffbe821cb9e42c4 already in progress
W20251026 08:52:57.777822 3707 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for c1e736aea2b54b378ffbe821cb9e42c4 already in progress
I20251026 08:52:58.147105 3835 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20251026 08:52:58.147562 4606 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:58.149300 4300 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:52:58.149868 4545 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:52:58.153062 3760 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 28. Preceding OpId from leader: term: 4 index: 29. (index mismatch)
I20251026 08:52:58.154649 4545 raft_consensus.cc:993] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower b983c9a7c2924030a396232a4ad46572 to start an election
I20251026 08:52:58.154743 4545 raft_consensus.cc:1081] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Signalling peer b983c9a7c2924030a396232a4ad46572 to start an election
I20251026 08:52:58.154764 4606 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:58.155701 4299 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4"
dest_uuid: "b983c9a7c2924030a396232a4ad46572"
from {username='slave'} at 127.0.0.1:52940
I20251026 08:52:58.157480 3910 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:52:58.158041 4615 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
W20251026 08:52:58.171114 3727 consensus_peers.cc:597] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:58.178067 4299 raft_consensus.cc:493] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:52:58.178207 4299 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:58.180470 4299 raft_consensus.cc:515] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.181401 4299 leader_election.cc:290] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [CANDIDATE]: Term 3 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:52:58.181757 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "b983c9a7c2924030a396232a4ad46572" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:52:58.182037 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e736aea2b54b378ffbe821cb9e42c4" candidate_uuid: "b983c9a7c2924030a396232a4ad46572" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:52:58.182219 3910 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:58.182742 3758 raft_consensus.cc:3055] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:52:58.182806 3758 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:52:58.182984 3758 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.183279 3758 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:52:58.184432 3910 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b983c9a7c2924030a396232a4ad46572 in term 3.
I20251026 08:52:58.184752 4265 leader_election.cc:304] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, b983c9a7c2924030a396232a4ad46572; no voters:
I20251026 08:52:58.185369 3758 raft_consensus.cc:2468] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b983c9a7c2924030a396232a4ad46572 in term 3.
I20251026 08:52:58.185614 4492 raft_consensus.cc:2804] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:52:58.186355 4492 raft_consensus.cc:697] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 LEADER]: Becoming Leader. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Running, Role: LEADER
I20251026 08:52:58.186570 4492 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.188537 3650 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 reported cstate change: term changed from 2 to 3, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to b983c9a7c2924030a396232a4ad46572 (127.0.103.70). New cstate: current_term: 3 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:58.605104 3910 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:52:58.605607 4492 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:52:58.609943 3758 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:52:58.611876 4633 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
W20251026 08:52:58.618009 4266 consensus_peers.cc:597] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:52:58.675877 4590 tablet_copy_client.cc:538] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:58.679281 4590 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap starting.
I20251026 08:52:58.707480 4593 tablet_copy_client.cc:538] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:58.710530 4593 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap starting.
I20251026 08:52:58.719863 4590 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=111 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:58.720201 4590 tablet_bootstrap.cc:492] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap complete.
I20251026 08:52:58.720446 4590 ts_tablet_manager.cc:1403] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: Time spent bootstrapping tablet: real 0.041s user 0.035s sys 0.001s
I20251026 08:52:58.720934 4590 raft_consensus.cc:359] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.721107 4590 raft_consensus.cc:740] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Initialized, Role: LEARNER
I20251026 08:52:58.721271 4590 consensus_queue.cc:260] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 2.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.721644 4590 ts_tablet_manager.cc:1434] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:52:58.722262 4236 tablet_copy_service.cc:342] P 49e9ac1b52f24abf8690af5a6924646c: Request end of tablet copy session 381d5756da8c4fa3ab5663d92bf76739-0409b48759d34a6fa9c86d88b809e093 received from {username='slave'} at 127.0.0.1:58254
I20251026 08:52:58.722393 4236 tablet_copy_service.cc:434] P 49e9ac1b52f24abf8690af5a6924646c: ending tablet copy session 381d5756da8c4fa3ab5663d92bf76739-0409b48759d34a6fa9c86d88b809e093 on tablet 0409b48759d34a6fa9c86d88b809e093 with peer 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:52:58.745167 4593 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=125 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:58.745466 4593 tablet_bootstrap.cc:492] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap complete.
I20251026 08:52:58.745676 4593 ts_tablet_manager.cc:1403] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.006s
I20251026 08:52:58.746325 4593 raft_consensus.cc:359] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:58.746500 4593 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Initialized, Role: LEARNER
I20251026 08:52:58.746687 4593 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:52:58.747236 4593 ts_tablet_manager.cc:1434] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20251026 08:52:58.747994 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session 49e9ac1b52f24abf8690af5a6924646c-b1edddc92bb0463b9ed275da3c7154ef received from {username='slave'} at 127.0.0.1:47636
I20251026 08:52:58.748154 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session 49e9ac1b52f24abf8690af5a6924646c-b1edddc92bb0463b9ed275da3c7154ef on tablet b1edddc92bb0463b9ed275da3c7154ef with peer 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:52:58.793684 4597 tablet_copy_client.cc:538] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:58.796584 4597 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap starting.
I20251026 08:52:58.802552 4599 tablet_copy_client.cc:538] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:58.805445 4599 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: Bootstrap starting.
I20251026 08:52:58.831229 4597 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap replayed 1/1 log segments. Stats: ops{read=32 overwritten=0 applied=32 ignored=0} inserts{seen=86 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:58.831537 4597 tablet_bootstrap.cc:492] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap complete.
I20251026 08:52:58.831790 4597 ts_tablet_manager.cc:1403] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.011s
I20251026 08:52:58.832484 4597 raft_consensus.cc:359] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.832659 4597 raft_consensus.cc:740] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Initialized, Role: LEARNER
I20251026 08:52:58.832857 4597 consensus_queue.cc:260] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 32, Last appended: 3.32, Last appended by leader: 32, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.833935 4597 ts_tablet_manager.cc:1434] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251026 08:52:58.834482 3846 tablet_copy_service.cc:342] P 96cd069722b34c1eab8ae9a2f953dd7a: Request end of tablet copy session 381d5756da8c4fa3ab5663d92bf76739-5f3c583b37d74efe9630e3723a2c313e received from {username='slave'} at 127.0.0.1:42606
I20251026 08:52:58.834625 3846 tablet_copy_service.cc:434] P 96cd069722b34c1eab8ae9a2f953dd7a: ending tablet copy session 381d5756da8c4fa3ab5663d92bf76739-5f3c583b37d74efe9630e3723a2c313e on tablet 5f3c583b37d74efe9630e3723a2c313e with peer 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:52:58.840713 4599 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: Bootstrap replayed 1/1 log segments. Stats: ops{read=28 overwritten=0 applied=28 ignored=0} inserts{seen=115 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:58.840989 4599 tablet_bootstrap.cc:492] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: Bootstrap complete.
I20251026 08:52:58.841207 4599 ts_tablet_manager.cc:1403] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.006s
I20251026 08:52:58.841846 4599 raft_consensus.cc:359] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:58.842129 4599 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Initialized, Role: LEARNER
I20251026 08:52:58.842752 4599 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 28, Last appended: 2.28, Last appended by leader: 28, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:58.843478 4599 ts_tablet_manager.cc:1434] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:52:58.843997 3846 tablet_copy_service.cc:342] P 96cd069722b34c1eab8ae9a2f953dd7a: Request end of tablet copy session b983c9a7c2924030a396232a4ad46572-6dd57692a7704c9690a892319e4bf446 received from {username='slave'} at 127.0.0.1:42620
I20251026 08:52:58.844115 3846 tablet_copy_service.cc:434] P 96cd069722b34c1eab8ae9a2f953dd7a: ending tablet copy session b983c9a7c2924030a396232a4ad46572-6dd57692a7704c9690a892319e4bf446 on tablet 6dd57692a7704c9690a892319e4bf446 with peer b983c9a7c2924030a396232a4ad46572
I20251026 08:52:58.897310 4601 tablet_copy_client.cc:538] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:58.900291 4601 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap starting.
I20251026 08:52:58.923383 4603 tablet_copy_client.cc:538] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:52:58.926126 4603 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: Bootstrap starting.
I20251026 08:52:58.931449 4601 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=97 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:58.931752 4601 tablet_bootstrap.cc:492] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap complete.
I20251026 08:52:58.932019 4601 ts_tablet_manager.cc:1403] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: Time spent bootstrapping tablet: real 0.032s user 0.025s sys 0.008s
I20251026 08:52:58.932610 4601 raft_consensus.cc:359] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.932781 4601 raft_consensus.cc:740] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Initialized, Role: LEARNER
I20251026 08:52:58.933007 4601 consensus_queue.cc:260] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 1.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:52:58.933576 4601 ts_tablet_manager.cc:1434] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 08:52:58.934113 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session 381d5756da8c4fa3ab5663d92bf76739-c1e736aea2b54b378ffbe821cb9e42c4 received from {username='slave'} at 127.0.0.1:47624
I20251026 08:52:58.934229 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session 381d5756da8c4fa3ab5663d92bf76739-c1e736aea2b54b378ffbe821cb9e42c4 on tablet c1e736aea2b54b378ffbe821cb9e42c4 with peer 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:52:58.949409 4300 raft_consensus.cc:1217] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20251026 08:52:58.965821 4603 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=101 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:52:58.966168 4603 tablet_bootstrap.cc:492] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: Bootstrap complete.
I20251026 08:52:58.966434 4603 ts_tablet_manager.cc:1403] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.011s
I20251026 08:52:58.967069 4603 raft_consensus.cc:359] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:58.967223 4603 raft_consensus.cc:740] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Initialized, Role: LEARNER
I20251026 08:52:58.967404 4603 consensus_queue.cc:260] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:52:58.967883 4603 ts_tablet_manager.cc:1434] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:52:58.968537 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session b983c9a7c2924030a396232a4ad46572-d0a49b69e0a94ae9827e91c43006c2d5 received from {username='slave'} at 127.0.0.1:47646
I20251026 08:52:58.968689 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session b983c9a7c2924030a396232a4ad46572-d0a49b69e0a94ae9827e91c43006c2d5 on tablet d0a49b69e0a94ae9827e91c43006c2d5 with peer b983c9a7c2924030a396232a4ad46572
I20251026 08:52:59.022292 4151 raft_consensus.cc:1217] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.31->[3.32-3.32] Dedup: 3.32->[]
I20251026 08:52:59.108145 4226 raft_consensus.cc:1217] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.27->[2.28-2.28] Dedup: 2.28->[]
I20251026 08:52:59.121825 4151 raft_consensus.cc:3060] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 1 LEARNER]: Advancing to term 3
I20251026 08:52:59.124024 4151 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 1 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:52:59.124511 4633 consensus_queue.cc:1050] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:52:59.134794 4150 raft_consensus.cc:1217] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.30->[2.31-2.31] Dedup: 2.31->[]
I20251026 08:52:59.139237 4633 raft_consensus.cc:1064] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: attempting to promote NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 to VOTER
I20251026 08:52:59.140130 4633 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 31, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:59.142778 3910 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:52:59.142948 3758 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:52:59.143263 4633 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.143445 4151 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:52:59.143638 4643 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.143952 4644 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.145645 4643 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 30 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.146145 3910 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.147058 4151 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.150326 3758 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 30 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.151165 3650 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: config changed from index 30 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.157739 4300 consensus_queue.cc:237] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 31, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:59.160391 3910 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20251026 08:52:59.160815 4151 raft_consensus.cc:1275] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20251026 08:52:59.160995 4633 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251026 08:52:59.162357 4633 consensus_queue.cc:1048] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251026 08:52:59.162889 4633 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.163350 3907 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.164775 4151 raft_consensus.cc:2955] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.166918 3656 catalog_manager.cc:5649] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 33 to 34, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New cstate: current_term: 3 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.167596 3701 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c1e736aea2b54b378ffbe821cb9e42c4 with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:59.170086 3748 tablet_service.cc:1552] Processing DeleteTablet for tablet c1e736aea2b54b378ffbe821cb9e42c4 with delete_type TABLET_DATA_TOMBSTONED (TS e7be295d3ba247a1a4475fa020d1ade1 not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:59.170701 4649 tablet_replica.cc:333] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:52:59.170886 4649 raft_consensus.cc:2243] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:59.171219 4649 raft_consensus.cc:2272] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:59.172001 4649 ts_tablet_manager.cc:1916] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:59.176358 4649 ts_tablet_manager.cc:1929] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20251026 08:52:59.176472 4649 log.cc:1199] T c1e736aea2b54b378ffbe821cb9e42c4 P e7be295d3ba247a1a4475fa020d1ade1: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/wals/c1e736aea2b54b378ffbe821cb9e42c4
I20251026 08:52:59.177158 3646 catalog_manager.cc:4985] TS e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): tablet c1e736aea2b54b378ffbe821cb9e42c4 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:59.329262 4550 raft_consensus.cc:1064] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
I20251026 08:52:59.330171 4550 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:59.332036 4300 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:59.332074 3758 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:59.332088 3907 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:59.332484 4651 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:59.332758 4550 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:59.333009 4650 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:59.337134 4550 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.340283 4300 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.343747 3758 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.345108 3657 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 reported cstate change: config changed from index 28 to 29, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.349472 3910 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.350040 3832 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:52:59.351754 4300 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:59.352368 4651 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:59.352595 3758 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:59.353264 4651 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:59.355312 4653 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.355856 4300 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.356462 3758 raft_consensus.cc:2955] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.359175 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 6dd57692a7704c9690a892319e4bf446 with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:59.360677 3657 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 reported cstate change: config changed from index 29 to 30, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New cstate: current_term: 2 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:52:59.363570 3898 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dd57692a7704c9690a892319e4bf446 with delete_type TABLET_DATA_TOMBSTONED (TS 53f35e5629a94c8f81ad198ebc4df6ae not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:47574
I20251026 08:52:59.364122 4657 tablet_replica.cc:333] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:52:59.364347 4657 raft_consensus.cc:2243] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:59.364537 4657 raft_consensus.cc:2272] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:59.365173 4657 ts_tablet_manager.cc:1916] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:59.370308 4657 ts_tablet_manager.cc:1929] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20251026 08:52:59.370438 4657 log.cc:1199] T 6dd57692a7704c9690a892319e4bf446 P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/wals/6dd57692a7704c9690a892319e4bf446
I20251026 08:52:59.371029 3645 catalog_manager.cc:4985] TS 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): tablet 6dd57692a7704c9690a892319e4bf446 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:59.441555 4300 raft_consensus.cc:1217] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20251026 08:52:59.615927 4658 raft_consensus.cc:1064] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: attempting to promote NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 to VOTER
I20251026 08:52:59.616554 4658 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 29, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:59.617625 4653 raft_consensus.cc:1064] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: attempting to promote NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 to VOTER
I20251026 08:52:59.618367 4653 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 28, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:59.619539 3832 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:52:59.619625 4151 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:52:59.619910 3758 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:52:59.620298 4619 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:52:59.620680 4619 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:52:59.621018 4619 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:52:59.621345 4151 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:52:59.621598 3907 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:52:59.621940 4651 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.621938 4300 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:52:59.622689 4651 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.622987 4650 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.633235 4658 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.637601 3832 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.637717 3758 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.643286 3650 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 31 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.644416 4149 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.651297 4226 consensus_queue.cc:237] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 2.32, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:59.653074 3760 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20251026 08:52:59.653360 4650 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.653769 4151 raft_consensus.cc:1275] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 32. Preceding OpId from leader: term: 2 index: 33. (index mismatch)
I20251026 08:52:59.654712 4619 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.654806 4151 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.655058 4619 consensus_queue.cc:1048] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.656347 4659 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEADER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.656816 3758 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.657631 3907 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.660048 3656 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: config changed from index 32 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.662181 3653 catalog_manager.cc:5649] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.664213 3705 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0409b48759d34a6fa9c86d88b809e093 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:59.665865 3822 tablet_service.cc:1552] Processing DeleteTablet for tablet 0409b48759d34a6fa9c86d88b809e093 with delete_type TABLET_DATA_TOMBSTONED (TS 96cd069722b34c1eab8ae9a2f953dd7a not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:60834
I20251026 08:52:59.666070 3832 consensus_queue.cc:237] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 3.33, Last appended by leader: 28, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:52:59.666496 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:52:59.669576 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:52:59.669744 3907 raft_consensus.cc:606] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:52:59.669922 4299 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20251026 08:52:59.670756 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:52:59.670904 3835 raft_consensus.cc:606] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:52:59.671269 4653 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:52:59.671922 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 08:52:59.672144 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:52:59.672513 4299 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.675145 4670 tablet_replica.cc:333] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:52:59.675309 4670 raft_consensus.cc:2243] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:59.675379 4669 raft_consensus.cc:1064] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: attempting to promote NON_VOTER 49e9ac1b52f24abf8690af5a6924646c to VOTER
I20251026 08:52:59.675900 4670 raft_consensus.cc:2272] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:59.676168 4151 raft_consensus.cc:1275] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 33. Preceding OpId from leader: term: 3 index: 34. (index mismatch)
I20251026 08:52:59.677413 4550 consensus_queue.cc:1048] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251026 08:52:59.677290 4650 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.678584 4670 ts_tablet_manager.cc:1916] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:59.679157 4149 raft_consensus.cc:2955] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.33: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.680564 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 5f3c583b37d74efe9630e3723a2c313e with cas_config_opid_index 33: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:59.676676 4669 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 28, Committed index: 28, Last appended: 2.28, Last appended by leader: 26, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:59.681231 3657 catalog_manager.cc:5649] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 33 to 34, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:59.683869 4299 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.680305 4151 raft_consensus.cc:2955] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Committing config change with OpId 3.34: config changed from index 33 to 34, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 34 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:52:59.685710 3898 tablet_service.cc:1552] Processing DeleteTablet for tablet 5f3c583b37d74efe9630e3723a2c313e with delete_type TABLET_DATA_TOMBSTONED (TS 53f35e5629a94c8f81ad198ebc4df6ae not found in new config with opid_index 34) from {username='slave'} at 127.0.0.1:47574
I20251026 08:52:59.685997 4657 tablet_replica.cc:333] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:52:59.686197 4657 raft_consensus.cc:2243] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:59.686391 4657 raft_consensus.cc:2272] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:59.687117 4657 ts_tablet_manager.cc:1916] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:59.688788 4670 ts_tablet_manager.cc:1929] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.32
I20251026 08:52:59.688915 4670 log.cc:1199] T 0409b48759d34a6fa9c86d88b809e093 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/wals/0409b48759d34a6fa9c86d88b809e093
I20251026 08:52:59.689543 3645 catalog_manager.cc:4985] TS 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): tablet 0409b48759d34a6fa9c86d88b809e093 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:59.690378 4225 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:59.690502 3758 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:59.690886 3832 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 28. Preceding OpId from leader: term: 2 index: 29. (index mismatch)
I20251026 08:52:59.690949 4673 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:59.693797 4657 ts_tablet_manager.cc:1929] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.33
I20251026 08:52:59.693924 4657 log.cc:1199] T 5f3c583b37d74efe9630e3723a2c313e P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/wals/5f3c583b37d74efe9630e3723a2c313e
I20251026 08:52:59.694939 4674 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:59.695286 4669 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 29, Last known committed idx: 28, Time since last communication: 0.000s
I20251026 08:52:59.695281 3645 catalog_manager.cc:4985] TS 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): tablet 5f3c583b37d74efe9630e3723a2c313e (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:52:59.704167 4607 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.704727 3835 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.707535 3758 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.708695 4225 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.29: config changed from index 28 to 29, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.708931 3650 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 28 to 29, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:52:59.715655 3910 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:52:59.716907 4225 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:59.717249 3835 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:52:59.717280 4669 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:59.717677 4669 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:52:59.719105 4671 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.719609 4225 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.722465 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet b1edddc92bb0463b9ed275da3c7154ef with cas_config_opid_index 29: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:52:59.722815 3650 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: config changed from index 29 to 30, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.722808 3835 raft_consensus.cc:2955] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:52:59.726263 3748 tablet_service.cc:1552] Processing DeleteTablet for tablet b1edddc92bb0463b9ed275da3c7154ef with delete_type TABLET_DATA_TOMBSTONED (TS e7be295d3ba247a1a4475fa020d1ade1 not found in new config with opid_index 30) from {username='slave'} at 127.0.0.1:51950
I20251026 08:52:59.726845 4678 tablet_replica.cc:333] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:52:59.727156 4678 raft_consensus.cc:2243] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:52:59.727550 4678 raft_consensus.cc:2272] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:52:59.728279 4678 ts_tablet_manager.cc:1916] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:52:59.732741 4678 ts_tablet_manager.cc:1929] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.29
I20251026 08:52:59.732856 4678 log.cc:1199] T b1edddc92bb0463b9ed275da3c7154ef P e7be295d3ba247a1a4475fa020d1ade1: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/wals/b1edddc92bb0463b9ed275da3c7154ef
I20251026 08:52:59.733402 3646 catalog_manager.cc:4985] TS e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): tablet b1edddc92bb0463b9ed275da3c7154ef (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:53:00.061604 4673 raft_consensus.cc:1064] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
I20251026 08:53:00.062557 4673 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:00.064587 4299 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEARNER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:53:00.064584 3835 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:53:00.064587 4151 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:53:00.065223 4674 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:00.065531 4607 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:00.065770 4669 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:00.069432 4674 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.070001 4151 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.072670 3835 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.074291 3657 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: config changed from index 30 to 31, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.074513 4299 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.092988 3910 consensus_queue.cc:237] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:00.094355 4149 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:53:00.094384 4299 raft_consensus.cc:1275] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:53:00.094775 4607 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:00.095042 4673 consensus_queue.cc:1048] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:00.096680 4673 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.098044 4299 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.099802 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet d0a49b69e0a94ae9827e91c43006c2d5 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:00.099926 4151 raft_consensus.cc:2955] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:00.100859 3653 catalog_manager.cc:5649] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:00.104831 3822 tablet_service.cc:1552] Processing DeleteTablet for tablet d0a49b69e0a94ae9827e91c43006c2d5 with delete_type TABLET_DATA_TOMBSTONED (TS 96cd069722b34c1eab8ae9a2f953dd7a not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:60834
I20251026 08:53:00.105130 4670 tablet_replica.cc:333] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:00.105312 4670 raft_consensus.cc:2243] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:00.105553 4670 raft_consensus.cc:2272] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:00.106278 4670 ts_tablet_manager.cc:1916] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:00.109831 4670 ts_tablet_manager.cc:1929] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20251026 08:53:00.109916 4670 log.cc:1199] T d0a49b69e0a94ae9827e91c43006c2d5 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/wals/d0a49b69e0a94ae9827e91c43006c2d5
I20251026 08:53:00.110550 3645 catalog_manager.cc:4985] TS 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): tablet d0a49b69e0a94ae9827e91c43006c2d5 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:53:00.151923 4651 raft_consensus.cc:993] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:53:00.152060 4651 raft_consensus.cc:1081] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:53:00.152511 3907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:53:00.152647 3907 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:00.152717 3907 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:00.154273 3907 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:53:00.154690 3907 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 5 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:53:00.155052 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:53:00.155058 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:00.155225 3758 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:00.155304 3835 raft_consensus.cc:3055] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Stepping down as leader of term 4
I20251026 08:53:00.155413 3835 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:00.155619 3835 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:53:00.155874 3835 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:00.156874 3758 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 5.
I20251026 08:53:00.157179 3876 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:00.157424 4607 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Leader election won for term 5
I20251026 08:53:00.157635 4607 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:00.157667 3835 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 5.
I20251026 08:53:00.158036 4607 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } }
I20251026 08:53:00.160048 3650 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 4 to 5, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 5 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:00.181661 4673 raft_consensus.cc:993] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:00.181763 4673 raft_consensus.cc:1081] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:00.182199 3758 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51954
I20251026 08:53:00.182323 3758 raft_consensus.cc:493] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:00.182415 3758 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:00.183804 3758 raft_consensus.cc:515] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:53:00.184293 3758 leader_election.cc:290] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 5 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913)
I20251026 08:53:00.184633 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:53:00.184739 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3d3135b8b6fe4dff81329c307264ea9f" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 5 candidate_status { last_received { term: 4 index: 29 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:00.184794 3907 raft_consensus.cc:3055] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Stepping down as leader of term 4
I20251026 08:53:00.184898 3907 raft_consensus.cc:740] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:00.184928 3835 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:00.185096 3907 consensus_queue.cc:260] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:53:00.185350 3907 raft_consensus.cc:3060] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:00.186956 3835 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 5.
I20251026 08:53:00.187260 3726 leader_election.cc:304] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:00.187427 3907 raft_consensus.cc:2468] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 5.
I20251026 08:53:00.187500 4626 raft_consensus.cc:2804] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 08:53:00.187857 4626 raft_consensus.cc:697] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 5 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:00.188100 4626 consensus_queue.cc:237] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 4.29, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } }
I20251026 08:53:00.189914 3650 catalog_manager.cc:5649] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 4 to 5, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 5 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:00.582525 3835 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20251026 08:53:00.583019 4626 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:00.586419 3907 raft_consensus.cc:1275] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20251026 08:53:00.586931 4684 consensus_queue.cc:1048] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:00.625279 3835 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20251026 08:53:00.625733 4673 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:00.628481 3758 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 4 index: 29. Preceding OpId from leader: term: 5 index: 30. (index mismatch)
I20251026 08:53:00.628960 4694 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:01.114992 4151 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:53:01.116947 3835 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:53:01.117205 3907 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
W20251026 08:53:01.116964 4116 consensus_peers.cc:597] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b2c4bd03bfd4efd8aa306d8557fad3e. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:01.117476 4700 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:01.117663 4699 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:01.119969 4700 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEADER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.120625 3907 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.122825 3835 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Committing config change with OpId 3.31: config changed from index 29 to 31, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.124408 3650 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: config changed from index 29 to 31, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New cstate: current_term: 3 leader_uuid: "381d5756da8c4fa3ab5663d92bf76739" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:01.126734 4299 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:53:01.131606 3758 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:53:01.132220 4633 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:01.132477 3835 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:53:01.132833 4709 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:01.134608 4633 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.135038 3758 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.138080 3653 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 28 to 30, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New cstate: current_term: 2 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.142004 3907 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 5.30, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
W20251026 08:53:01.143899 4265 consensus_peers.cc:597] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 -> Peer 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723): Couldn't send request to peer 49e9ac1b52f24abf8690af5a6924646c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5f200f20b1741b4a2c6bc379b2ae8f5. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:01.144096 3758 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
W20251026 08:53:01.144508 3873 consensus_peers.cc:597] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae -> Peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541): Couldn't send request to peer b983c9a7c2924030a396232a4ad46572. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ee117517b8b4ae18fa61970868914a4. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:01.144685 4694 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:01.144970 3835 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 28 to 30, NON_VOTER 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } } }
I20251026 08:53:01.145927 3832 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 30. Preceding OpId from leader: term: 5 index: 31. (index mismatch)
I20251026 08:53:01.146366 4694 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:01.146929 4694 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:53:01.147301 3760 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:53:01.151784 3760 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 2.29, Last appended by leader: 26, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:53:01.152285 3656 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index -1 to 31, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New cstate: current_term: 5 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
I20251026 08:53:01.152732 3832 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Committing config change with OpId 5.31: config changed from index -1 to 31, NON_VOTER b983c9a7c2924030a396232a4ad46572 (127.0.103.70) added. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } } }
W20251026 08:53:01.154953 3727 consensus_peers.cc:597] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85fe4797e56f4170876f76414e23628f. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:01.155098 4225 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:53:01.155520 4703 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:01.155872 3907 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:53:01.156291 4703 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:01.159018 4704 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:53:01.159508 4225 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:53:01.162492 3656 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:01.163925 3907 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Committing config change with OpId 2.30: config changed from index 29 to 30, NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) added. New config: { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } } }
I20251026 08:53:01.572394 4722 ts_tablet_manager.cc:933] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: Initiating tablet copy from peer b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:53:01.572894 4722 tablet_copy_client.cc:323] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.70:46541
I20251026 08:53:01.578097 4310 tablet_copy_service.cc:140] P b983c9a7c2924030a396232a4ad46572: Received BeginTabletCopySession request for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 from peer 49e9ac1b52f24abf8690af5a6924646c ({username='slave'} at 127.0.0.1:40214)
I20251026 08:53:01.578217 4310 tablet_copy_service.cc:161] P b983c9a7c2924030a396232a4ad46572: Beginning new tablet copy session on tablet c5f200f20b1741b4a2c6bc379b2ae8f5 from peer 49e9ac1b52f24abf8690af5a6924646c at {username='slave'} at 127.0.0.1:40214: session id = 49e9ac1b52f24abf8690af5a6924646c-c5f200f20b1741b4a2c6bc379b2ae8f5
I20251026 08:53:01.580538 4310 tablet_copy_source_session.cc:215] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:01.581501 4722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f200f20b1741b4a2c6bc379b2ae8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:01.585570 4722 tablet_copy_client.cc:806] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:01.585800 4722 tablet_copy_client.cc:670] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:01.586910 4723 tablet_copy_client.cc:1025] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Injecting latency into file download: 2000
I20251026 08:53:01.591395 4726 ts_tablet_manager.cc:933] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: Initiating tablet copy from peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881)
I20251026 08:53:01.591879 4726 tablet_copy_client.cc:323] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.68:33881
I20251026 08:53:01.592374 4161 tablet_copy_service.cc:140] P 381d5756da8c4fa3ab5663d92bf76739: Received BeginTabletCopySession request for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e from peer 49e9ac1b52f24abf8690af5a6924646c ({username='slave'} at 127.0.0.1:33752)
I20251026 08:53:01.592518 4161 tablet_copy_service.cc:161] P 381d5756da8c4fa3ab5663d92bf76739: Beginning new tablet copy session on tablet 0b2c4bd03bfd4efd8aa306d8557fad3e from peer 49e9ac1b52f24abf8690af5a6924646c at {username='slave'} at 127.0.0.1:33752: session id = 49e9ac1b52f24abf8690af5a6924646c-0b2c4bd03bfd4efd8aa306d8557fad3e
I20251026 08:53:01.594794 4161 tablet_copy_source_session.cc:215] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:01.596184 4726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:01.600906 4726 tablet_copy_client.cc:806] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:01.601100 4726 tablet_copy_client.cc:670] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:01.602212 4727 tablet_copy_client.cc:1025] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Injecting latency into file download: 2000
I20251026 08:53:01.626579 4728 ts_tablet_manager.cc:933] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: Initiating tablet copy from peer 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:53:01.627051 4728 tablet_copy_client.cc:323] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:33171
I20251026 08:53:01.627503 3920 tablet_copy_service.cc:140] P 53f35e5629a94c8f81ad198ebc4df6ae: Received BeginTabletCopySession request for tablet 9ee117517b8b4ae18fa61970868914a4 from peer b983c9a7c2924030a396232a4ad46572 ({username='slave'} at 127.0.0.1:47646)
I20251026 08:53:01.627611 3920 tablet_copy_service.cc:161] P 53f35e5629a94c8f81ad198ebc4df6ae: Beginning new tablet copy session on tablet 9ee117517b8b4ae18fa61970868914a4 from peer b983c9a7c2924030a396232a4ad46572 at {username='slave'} at 127.0.0.1:47646: session id = b983c9a7c2924030a396232a4ad46572-9ee117517b8b4ae18fa61970868914a4
I20251026 08:53:01.629441 3920 tablet_copy_source_session.cc:215] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:01.630331 4728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee117517b8b4ae18fa61970868914a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:01.634686 4728 tablet_copy_client.cc:806] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:01.634867 4728 tablet_copy_client.cc:670] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:01.636008 4729 tablet_copy_client.cc:1025] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Injecting latency into file download: 2000
I20251026 08:53:01.672461 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:01.672664 4730 ts_tablet_manager.cc:933] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: Initiating tablet copy from peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515)
I20251026 08:53:01.673146 4730 tablet_copy_client.cc:323] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:39515
I20251026 08:53:01.675112 3760 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "85fe4797e56f4170876f76414e23628f"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "49e9ac1b52f24abf8690af5a6924646c"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:53:01.675271 3760 raft_consensus.cc:606] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Received request to transfer leadership to TS 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:53:01.676076 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:53:01.676213 3907 raft_consensus.cc:606] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:53:01.676555 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 08:53:01.676740 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:01.678733 3771 tablet_copy_service.cc:140] P e7be295d3ba247a1a4475fa020d1ade1: Received BeginTabletCopySession request for tablet 85fe4797e56f4170876f76414e23628f from peer 381d5756da8c4fa3ab5663d92bf76739 ({username='slave'} at 127.0.0.1:58224)
I20251026 08:53:01.678884 3771 tablet_copy_service.cc:161] P e7be295d3ba247a1a4475fa020d1ade1: Beginning new tablet copy session on tablet 85fe4797e56f4170876f76414e23628f from peer 381d5756da8c4fa3ab5663d92bf76739 at {username='slave'} at 127.0.0.1:58224: session id = 381d5756da8c4fa3ab5663d92bf76739-85fe4797e56f4170876f76414e23628f
I20251026 08:53:01.681316 3771 tablet_copy_source_session.cc:215] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:01.682286 4730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85fe4797e56f4170876f76414e23628f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:01.686592 4730 tablet_copy_client.cc:806] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:01.686780 4730 tablet_copy_client.cc:670] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:01.687944 4731 tablet_copy_client.cc:1025] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Injecting latency into file download: 2000
I20251026 08:53:01.716161 4690 raft_consensus.cc:993] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 49e9ac1b52f24abf8690af5a6924646c to start an election
I20251026 08:53:01.716265 4690 raft_consensus.cc:1081] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Signalling peer 49e9ac1b52f24abf8690af5a6924646c to start an election
I20251026 08:53:01.716718 4225 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "85fe4797e56f4170876f76414e23628f"
dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
from {username='slave'} at 127.0.0.1:47568
I20251026 08:53:01.716863 4225 raft_consensus.cc:493] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:01.716948 4225 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:01.719004 4225 raft_consensus.cc:515] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:53:01.719905 4225 leader_election.cc:290] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [CANDIDATE]: Term 3 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171)
I20251026 08:53:01.720006 3758 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "49e9ac1b52f24abf8690af5a6924646c" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:53:01.720065 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "85fe4797e56f4170876f76414e23628f" candidate_uuid: "49e9ac1b52f24abf8690af5a6924646c" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:53:01.720160 3758 raft_consensus.cc:3055] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:53:01.720239 3758 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:01.720175 3907 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:01.720471 3758 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:53:01.720793 3758 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:01.721979 3907 raft_consensus.cc:2468] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e9ac1b52f24abf8690af5a6924646c in term 3.
I20251026 08:53:01.722280 3758 raft_consensus.cc:2468] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49e9ac1b52f24abf8690af5a6924646c in term 3.
I20251026 08:53:01.722591 4192 leader_election.cc:304] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9ac1b52f24abf8690af5a6924646c, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:01.722776 4668 raft_consensus.cc:2804] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:53:01.723102 4668 raft_consensus.cc:697] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEADER]: Becoming Leader. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Running, Role: LEADER
I20251026 08:53:01.723304 4668 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:53:01.725368 3656 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: term changed from 2 to 3, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69). New cstate: current_term: 3 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:02.038576 4717 raft_consensus.cc:993] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:02.038753 4717 raft_consensus.cc:1081] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:02.039289 3832 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60842
I20251026 08:53:02.039470 3832 raft_consensus.cc:493] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:02.039564 3832 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:02.041895 3832 raft_consensus.cc:515] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:02.042826 3832 leader_election.cc:290] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 3 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:53:02.043165 3910 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:53:02.043344 3910 raft_consensus.cc:3055] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:53:02.043421 3910 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:02.043704 3910 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:02.044034 3910 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:02.046784 3910 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 3.
I20251026 08:53:02.047111 3801 leader_election.cc:304] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:53:02.047348 4686 raft_consensus.cc:2804] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:53:02.047811 4686 raft_consensus.cc:697] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:02.048139 4686 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:02.051323 3656 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 2 to 3, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 3 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:02.060461 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
I20251026 08:53:02.060629 4226 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:02.062916 4226 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 3.
I20251026 08:53:02.243005 3758 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:53:02.243459 4668 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:02.248171 3910 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
W20251026 08:53:02.248603 4192 consensus_peers.cc:597] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c -> Peer 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68:33881): Couldn't send request to peer 381d5756da8c4fa3ab5663d92bf76739. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:02.248710 4752 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:02.424697 3910 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:53:02.425163 4686 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:02.431309 4226 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:53:02.431959 4758 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:03.587216 4722 tablet_copy_client.cc:538] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:03.589933 4722 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap starting.
I20251026 08:53:03.602576 4726 tablet_copy_client.cc:538] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:03.605530 4726 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap starting.
I20251026 08:53:03.625972 4722 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:03.626307 4722 tablet_bootstrap.cc:492] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap complete.
I20251026 08:53:03.626544 4722 ts_tablet_manager.cc:1403] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: Time spent bootstrapping tablet: real 0.037s user 0.031s sys 0.004s
I20251026 08:53:03.627091 4722 raft_consensus.cc:359] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:53:03.627249 4722 raft_consensus.cc:740] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Initialized, Role: LEARNER
I20251026 08:53:03.627427 4722 consensus_queue.cc:260] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:53:03.628746 4722 ts_tablet_manager.cc:1434] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:03.629539 4310 tablet_copy_service.cc:342] P b983c9a7c2924030a396232a4ad46572: Request end of tablet copy session 49e9ac1b52f24abf8690af5a6924646c-c5f200f20b1741b4a2c6bc379b2ae8f5 received from {username='slave'} at 127.0.0.1:40214
I20251026 08:53:03.629657 4310 tablet_copy_service.cc:434] P b983c9a7c2924030a396232a4ad46572: ending tablet copy session 49e9ac1b52f24abf8690af5a6924646c-c5f200f20b1741b4a2c6bc379b2ae8f5 on tablet c5f200f20b1741b4a2c6bc379b2ae8f5 with peer 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:53:03.636340 4728 tablet_copy_client.cc:538] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:03.639524 4728 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: Bootstrap starting.
I20251026 08:53:03.642027 4726 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=103 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:03.642323 4726 tablet_bootstrap.cc:492] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: Bootstrap complete.
I20251026 08:53:03.642515 4726 ts_tablet_manager.cc:1403] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: Time spent bootstrapping tablet: real 0.037s user 0.037s sys 0.000s
I20251026 08:53:03.642987 4726 raft_consensus.cc:359] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:53:03.643129 4726 raft_consensus.cc:740] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 49e9ac1b52f24abf8690af5a6924646c, State: Initialized, Role: LEARNER
I20251026 08:53:03.643262 4726 consensus_queue.cc:260] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: NON_VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: true } }
I20251026 08:53:03.643693 4726 ts_tablet_manager.cc:1434] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:53:03.644223 4161 tablet_copy_service.cc:342] P 381d5756da8c4fa3ab5663d92bf76739: Request end of tablet copy session 49e9ac1b52f24abf8690af5a6924646c-0b2c4bd03bfd4efd8aa306d8557fad3e received from {username='slave'} at 127.0.0.1:33752
I20251026 08:53:03.644333 4161 tablet_copy_service.cc:434] P 381d5756da8c4fa3ab5663d92bf76739: ending tablet copy session 49e9ac1b52f24abf8690af5a6924646c-0b2c4bd03bfd4efd8aa306d8557fad3e on tablet 0b2c4bd03bfd4efd8aa306d8557fad3e with peer 49e9ac1b52f24abf8690af5a6924646c
I20251026 08:53:03.671687 4728 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: Bootstrap replayed 1/1 log segments. Stats: ops{read=31 overwritten=0 applied=31 ignored=0} inserts{seen=110 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:03.671968 4728 tablet_bootstrap.cc:492] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: Bootstrap complete.
I20251026 08:53:03.672170 4728 ts_tablet_manager.cc:1403] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: Time spent bootstrapping tablet: real 0.033s user 0.031s sys 0.001s
I20251026 08:53:03.672649 4728 raft_consensus.cc:359] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:53:03.672853 4728 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 LEARNER]: Becoming Follower/Learner. State: Replica: b983c9a7c2924030a396232a4ad46572, State: Initialized, Role: LEARNER
I20251026 08:53:03.673031 4728 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 31, Last appended: 5.31, Last appended by leader: 31, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: NON_VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: true } }
I20251026 08:53:03.673398 4728 ts_tablet_manager.cc:1434] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:53:03.673906 3920 tablet_copy_service.cc:342] P 53f35e5629a94c8f81ad198ebc4df6ae: Request end of tablet copy session b983c9a7c2924030a396232a4ad46572-9ee117517b8b4ae18fa61970868914a4 received from {username='slave'} at 127.0.0.1:47646
I20251026 08:53:03.674026 3920 tablet_copy_service.cc:434] P 53f35e5629a94c8f81ad198ebc4df6ae: ending tablet copy session b983c9a7c2924030a396232a4ad46572-9ee117517b8b4ae18fa61970868914a4 on tablet 9ee117517b8b4ae18fa61970868914a4 with peer b983c9a7c2924030a396232a4ad46572
I20251026 08:53:03.676992 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:03.680014 3832 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:53:03.680171 3832 raft_consensus.cc:606] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:53:03.680579 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:03.680778 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:03.688297 4730 tablet_copy_client.cc:538] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:03.690956 4730 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap starting.
I20251026 08:53:03.741225 4730 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=95 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:03.741588 4730 tablet_bootstrap.cc:492] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: Bootstrap complete.
I20251026 08:53:03.741875 4730 ts_tablet_manager.cc:1403] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: Time spent bootstrapping tablet: real 0.051s user 0.047s sys 0.004s
I20251026 08:53:03.742544 4730 raft_consensus.cc:359] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:53:03.742730 4730 raft_consensus.cc:740] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 381d5756da8c4fa3ab5663d92bf76739, State: Initialized, Role: LEARNER
I20251026 08:53:03.742887 4730 consensus_queue.cc:260] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: true } }
I20251026 08:53:03.743341 4730 ts_tablet_manager.cc:1434] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:53:03.743922 3771 tablet_copy_service.cc:342] P e7be295d3ba247a1a4475fa020d1ade1: Request end of tablet copy session 381d5756da8c4fa3ab5663d92bf76739-85fe4797e56f4170876f76414e23628f received from {username='slave'} at 127.0.0.1:58224
I20251026 08:53:03.744035 3771 tablet_copy_service.cc:434] P e7be295d3ba247a1a4475fa020d1ade1: ending tablet copy session 381d5756da8c4fa3ab5663d92bf76739-85fe4797e56f4170876f76414e23628f on tablet 85fe4797e56f4170876f76414e23628f with peer 381d5756da8c4fa3ab5663d92bf76739
I20251026 08:53:03.761966 4226 raft_consensus.cc:1217] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEARNER]: Deduplicated request from leader. Original: 3.30->[3.31-3.31] Dedup: 3.31->[]
I20251026 08:53:03.818624 4300 raft_consensus.cc:1217] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 LEARNER]: Deduplicated request from leader. Original: 5.30->[5.31-5.31] Dedup: 5.31->[]
I20251026 08:53:03.891717 4149 raft_consensus.cc:3060] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 2 LEARNER]: Advancing to term 3
I20251026 08:53:04.039633 4226 raft_consensus.cc:1217] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.29->[2.30-2.30] Dedup: 2.30->[]
I20251026 08:53:04.053259 4767 raft_consensus.cc:993] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:53:04.053447 4767 raft_consensus.cc:1081] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:53:04.054018 3907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:53:04.054172 3907 raft_consensus.cc:493] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:04.054251 3907 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:53:04.056089 3907 raft_consensus.cc:515] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.056710 3907 leader_election.cc:290] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 4 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:53:04.057091 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
I20251026 08:53:04.057116 3832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:04.057288 4226 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:53:04.057325 3832 raft_consensus.cc:3055] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Stepping down as leader of term 3
I20251026 08:53:04.057374 3832 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:04.057578 3832 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.057825 3832 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:53:04.059725 4226 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 4.
I20251026 08:53:04.060110 3875 leader_election.cc:304] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9ac1b52f24abf8690af5a6924646c, 53f35e5629a94c8f81ad198ebc4df6ae; no voters:
I20251026 08:53:04.060191 3832 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 4.
I20251026 08:53:04.060313 4762 raft_consensus.cc:2804] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Leader election won for term 4
I20251026 08:53:04.060746 4762 raft_consensus.cc:697] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:04.060973 4762 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.063235 3653 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 3 to 4, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 4 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:04.215924 4783 raft_consensus.cc:1064] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: attempting to promote NON_VOTER b983c9a7c2924030a396232a4ad46572 to VOTER
I20251026 08:53:04.216825 4783 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 5.31, Last appended by leader: 29, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:04.219132 3758 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20251026 08:53:04.219558 4300 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 LEARNER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20251026 08:53:04.219132 3832 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 31. Preceding OpId from leader: term: 5 index: 32. (index mismatch)
I20251026 08:53:04.220036 4783 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.220322 4777 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.220674 4762 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.224077 4762 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Committing config change with OpId 5.32: config changed from index 31 to 32, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.228858 3832 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.234941 3653 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 31 to 32, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New cstate: current_term: 5 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.235587 4300 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.241641 3758 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Committing config change with OpId 5.32: config changed from index 31 to 32, b983c9a7c2924030a396232a4ad46572 (127.0.103.70) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } attrs { replace: true } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.246157 3907 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 5.32, Last appended by leader: 29, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:04.247784 3832 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20251026 08:53:04.248126 4300 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 5 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20251026 08:53:04.248598 4762 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:04.248942 4762 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:04.249549 4790 raft_consensus.cc:1064] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: attempting to promote NON_VOTER 49e9ac1b52f24abf8690af5a6924646c to VOTER
I20251026 08:53:04.250255 4790 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 29, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.251230 4783 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.252103 4299 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.253978 3645 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 9ee117517b8b4ae18fa61970868914a4 with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:04.254397 3832 raft_consensus.cc:2955] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Committing config change with OpId 5.33: config changed from index 32 to 33, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.257289 3656 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 reported cstate change: config changed from index 32 to 33, VOTER e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) evicted. New cstate: current_term: 5 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } }
I20251026 08:53:04.260146 3748 tablet_service.cc:1552] Processing DeleteTablet for tablet 9ee117517b8b4ae18fa61970868914a4 with delete_type TABLET_DATA_TOMBSTONED (TS e7be295d3ba247a1a4475fa020d1ade1 not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:51950
I20251026 08:53:04.261000 4226 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEARNER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:53:04.261076 3907 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:53:04.261443 4791 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.261817 4791 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.270085 3832 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:53:04.270591 4790 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.273629 4790 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.274562 3832 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.278048 4226 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.278527 3650 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: config changed from index 31 to 32, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "381d5756da8c4fa3ab5663d92bf76739" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.278834 4793 tablet_replica.cc:333] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:04.279053 4793 raft_consensus.cc:2243] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:04.279217 4793 raft_consensus.cc:2272] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:04.280136 4793 ts_tablet_manager.cc:1916] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:04.282343 3907 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 31 to 32, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.285796 4149 consensus_queue.cc:237] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 29, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.286562 4793 ts_tablet_manager.cc:1929] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 5.32
I20251026 08:53:04.286692 4793 log.cc:1199] T 9ee117517b8b4ae18fa61970868914a4 P e7be295d3ba247a1a4475fa020d1ade1: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-0-root/wals/9ee117517b8b4ae18fa61970868914a4
I20251026 08:53:04.287241 4225 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:53:04.287335 3646 catalog_manager.cc:4985] TS e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): tablet 9ee117517b8b4ae18fa61970868914a4 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:53:04.287937 3907 raft_consensus.cc:1275] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 381d5756da8c4fa3ab5663d92bf76739: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:53:04.287907 4784 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:04.288488 4784 consensus_queue.cc:1048] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:04.289928 4790 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.290479 4225 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.290870 3907 raft_consensus.cc:2955] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.293300 3706 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:04.294173 3653 catalog_manager.cc:5649] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 reported cstate change: config changed from index 32 to 33, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New cstate: current_term: 3 leader_uuid: "381d5756da8c4fa3ab5663d92bf76739" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:04.297662 3822 tablet_service.cc:1552] Processing DeleteTablet for tablet 0b2c4bd03bfd4efd8aa306d8557fad3e with delete_type TABLET_DATA_TOMBSTONED (TS 96cd069722b34c1eab8ae9a2f953dd7a not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:60834
I20251026 08:53:04.298228 4797 tablet_replica.cc:333] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:04.298465 4797 raft_consensus.cc:2243] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:04.298655 4797 raft_consensus.cc:2272] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:04.299415 4797 ts_tablet_manager.cc:1916] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:04.303998 4797 ts_tablet_manager.cc:1929] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20251026 08:53:04.304122 4797 log.cc:1199] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/wals/0b2c4bd03bfd4efd8aa306d8557fad3e
I20251026 08:53:04.304710 3645 catalog_manager.cc:4985] TS 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): tablet 0b2c4bd03bfd4efd8aa306d8557fad3e (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:53:04.443658 4799 raft_consensus.cc:1064] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: attempting to promote NON_VOTER 381d5756da8c4fa3ab5663d92bf76739 to VOTER
I20251026 08:53:04.444567 4799 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 30, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:53:04.446455 4149 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEARNER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:53:04.446470 3758 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:53:04.446550 3907 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 3 index: 32. (index mismatch)
I20251026 08:53:04.447062 4771 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.447319 4799 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.447515 4798 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.450917 4799 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEADER]: Committing config change with OpId 3.32: config changed from index 30 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.451339 3907 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.454831 3656 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: config changed from index 30 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.455824 4151 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.463582 4804 raft_consensus.cc:1064] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: attempting to promote NON_VOTER 49e9ac1b52f24abf8690af5a6924646c to VOTER
I20251026 08:53:04.464306 4804 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 28, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.465446 3758 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Committing config change with OpId 3.32: config changed from index 30 to 32, 381d5756da8c4fa3ab5663d92bf76739 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } attrs { replace: true } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.466596 3832 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:53:04.466787 4226 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEARNER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:53:04.466890 3760 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 2 index: 31. (index mismatch)
I20251026 08:53:04.467300 4761 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:04.467785 4761 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:04.467890 4225 consensus_queue.cc:237] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 3.32, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } }
I20251026 08:53:04.468180 4761 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:04.470278 4151 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:53:04.471053 4798 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:04.471650 3754 raft_consensus.cc:1275] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Refusing update from remote peer 49e9ac1b52f24abf8690af5a6924646c: Log matching property violated. Preceding OpId in replica: term: 3 index: 32. Preceding OpId from leader: term: 3 index: 33. (index mismatch)
I20251026 08:53:04.472235 4798 consensus_queue.cc:1048] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:04.474290 4771 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEADER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.475039 4151 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.475435 3754 raft_consensus.cc:2955] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Committing config change with OpId 3.33: config changed from index 32 to 33, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New config: { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } } }
I20251026 08:53:04.476608 3705 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 85fe4797e56f4170876f76414e23628f with cas_config_opid_index 32: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:04.478222 3650 catalog_manager.cc:5649] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: config changed from index 32 to 33, VOTER 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) evicted. New cstate: current_term: 3 leader_uuid: "49e9ac1b52f24abf8690af5a6924646c" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381d5756da8c4fa3ab5663d92bf76739" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 33881 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:04.485195 3898 tablet_service.cc:1552] Processing DeleteTablet for tablet 85fe4797e56f4170876f76414e23628f with delete_type TABLET_DATA_TOMBSTONED (TS 53f35e5629a94c8f81ad198ebc4df6ae not found in new config with opid_index 33) from {username='slave'} at 127.0.0.1:47574
I20251026 08:53:04.486267 4225 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 4 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251026 08:53:04.486900 4789 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.496650 3835 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251026 08:53:04.497390 4789 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.510426 4809 tablet_replica.cc:333] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:04.510802 4809 raft_consensus.cc:2243] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:04.511294 4809 raft_consensus.cc:2272] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:04.513487 4809 ts_tablet_manager.cc:1916] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:04.516381 4803 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEADER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.518268 4224 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.524966 3835 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.525147 3650 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c reported cstate change: config changed from index 30 to 31, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.526947 4809 ts_tablet_manager.cc:1929] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.32
I20251026 08:53:04.527066 4809 log.cc:1199] T 85fe4797e56f4170876f76414e23628f P 53f35e5629a94c8f81ad198ebc4df6ae: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-2-root/wals/85fe4797e56f4170876f76414e23628f
I20251026 08:53:04.528009 3760 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.31: config changed from index 30 to 31, 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69) changed from NON_VOTER to VOTER. New config: { opid_index: 31 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } attrs { replace: true } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.528492 3645 catalog_manager.cc:4985] TS 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171): tablet 85fe4797e56f4170876f76414e23628f (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:53:04.533253 4299 consensus_queue.cc:237] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 2.31, Last appended by leader: 28, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:04.534587 4224 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:53:04.534999 4815 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.535804 3760 raft_consensus.cc:1275] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer b983c9a7c2924030a396232a4ad46572: Log matching property violated. Preceding OpId in replica: term: 2 index: 31. Preceding OpId from leader: term: 2 index: 32. (index mismatch)
I20251026 08:53:04.536377 4815 consensus_queue.cc:1048] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:04.536715 4804 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEADER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.539005 4224 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.539716 3701 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 with cas_config_opid_index 31: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:04.540716 3754 raft_consensus.cc:2955] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Committing config change with OpId 2.32: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New config: { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } } }
I20251026 08:53:04.542042 3653 catalog_manager.cc:5649] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 reported cstate change: config changed from index 31 to 32, VOTER 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) evicted. New cstate: current_term: 2 leader_uuid: "b983c9a7c2924030a396232a4ad46572" committed_config { opid_index: 32 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:04.544920 3822 tablet_service.cc:1552] Processing DeleteTablet for tablet c5f200f20b1741b4a2c6bc379b2ae8f5 with delete_type TABLET_DATA_TOMBSTONED (TS 96cd069722b34c1eab8ae9a2f953dd7a not found in new config with opid_index 32) from {username='slave'} at 127.0.0.1:60834
I20251026 08:53:04.545478 4797 tablet_replica.cc:333] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:04.545691 4797 raft_consensus.cc:2243] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:04.545879 4797 raft_consensus.cc:2272] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:04.546599 4797 ts_tablet_manager.cc:1916] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:04.550201 4797 ts_tablet_manager.cc:1929] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.31
I20251026 08:53:04.550316 4797 log.cc:1199] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 96cd069722b34c1eab8ae9a2f953dd7a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0/minicluster-data/ts-1-root/wals/c5f200f20b1741b4a2c6bc379b2ae8f5
I20251026 08:53:04.550827 3645 catalog_manager.cc:4985] TS 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913): tablet c5f200f20b1741b4a2c6bc379b2ae8f5 (table test-workload [id=ffe14307b6ea44a4bbb9d82ed3b82fcb]) successfully deleted
I20251026 08:53:05.681210 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:05.684278 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:53:05.684468 3907 raft_consensus.cc:606] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:53:05.684968 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:05.685184 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:06.124524 4762 raft_consensus.cc:993] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:06.124665 4762 raft_consensus.cc:1081] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:06.125202 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60842
I20251026 08:53:06.125386 3835 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:06.125526 3835 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Advancing to term 6
I20251026 08:53:06.127259 3835 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:06.127794 3835 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 6 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:53:06.128050 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:53:06.128204 3907 raft_consensus.cc:3055] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Stepping down as leader of term 5
I20251026 08:53:06.128285 3907 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:06.128211 4300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 6 candidate_status { last_received { term: 5 index: 33 } } ignore_live_leader: true dest_uuid: "b983c9a7c2924030a396232a4ad46572"
I20251026 08:53:06.128520 3907 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:06.128528 4300 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Advancing to term 6
I20251026 08:53:06.128834 3907 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Advancing to term 6
I20251026 08:53:06.130461 3907 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 6.
I20251026 08:53:06.130540 4300 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 6.
I20251026 08:53:06.130805 3801 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:53:06.131004 4767 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 FOLLOWER]: Leader election won for term 6
I20251026 08:53:06.131284 4767 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:06.131510 4767 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 5.33, Last appended by leader: 33, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:06.133296 3656 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 5 to 6, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 6 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:06.585477 3907 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 6 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20251026 08:53:06.585906 4767 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251026 08:53:06.590900 4300 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 6 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 5 index: 33. Preceding OpId from leader: term: 6 index: 34. (index mismatch)
I20251026 08:53:06.591312 4767 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 34, Last known committed idx: 33, Time since last communication: 0.000s
I20251026 08:53:07.685472 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:07.687783 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:53:07.687955 3835 raft_consensus.cc:606] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 LEADER]: Received request to transfer leadership to TS 53f35e5629a94c8f81ad198ebc4df6ae
I20251026 08:53:07.688341 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:07.688484 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:08.130292 4826 raft_consensus.cc:993] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:53:08.130414 4826 raft_consensus.cc:1081] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 LEADER]: Signalling peer 53f35e5629a94c8f81ad198ebc4df6ae to start an election
I20251026 08:53:08.130868 3907 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
from {username='slave'} at 127.0.0.1:47576
I20251026 08:53:08.131039 3907 raft_consensus.cc:493] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:08.131129 3907 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 6 FOLLOWER]: Advancing to term 7
I20251026 08:53:08.132784 3907 raft_consensus.cc:515] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:08.133317 3907 leader_election.cc:290] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 7 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:53:08.133587 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:08.133692 4300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9ee117517b8b4ae18fa61970868914a4" candidate_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" candidate_term: 7 candidate_status { last_received { term: 6 index: 34 } } ignore_live_leader: true dest_uuid: "b983c9a7c2924030a396232a4ad46572"
I20251026 08:53:08.133740 3835 raft_consensus.cc:3055] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 LEADER]: Stepping down as leader of term 6
I20251026 08:53:08.133828 3835 raft_consensus.cc:740] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:08.133841 4300 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 6 FOLLOWER]: Advancing to term 7
I20251026 08:53:08.134044 3835 consensus_queue.cc:260] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:08.134333 3835 raft_consensus.cc:3060] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 6 FOLLOWER]: Advancing to term 7
I20251026 08:53:08.135550 4300 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 7.
I20251026 08:53:08.135911 3873 leader_election.cc:304] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53f35e5629a94c8f81ad198ebc4df6ae, b983c9a7c2924030a396232a4ad46572; no voters:
I20251026 08:53:08.136125 4762 raft_consensus.cc:2804] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 7 FOLLOWER]: Leader election won for term 7
I20251026 08:53:08.136178 3835 raft_consensus.cc:2468] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53f35e5629a94c8f81ad198ebc4df6ae in term 7.
I20251026 08:53:08.136489 4762 raft_consensus.cc:697] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 7 LEADER]: Becoming Leader. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:08.136752 4762 consensus_queue.cc:237] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 6.34, Last appended by leader: 34, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:08.138859 3650 catalog_manager.cc:5649] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae reported cstate change: term changed from 6 to 7, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67). New cstate: current_term: 7 leader_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" committed_config { opid_index: 33 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:08.660086 4300 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 7 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20251026 08:53:08.660573 4762 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251026 08:53:08.665370 3835 raft_consensus.cc:1275] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 7 FOLLOWER]: Refusing update from remote peer 53f35e5629a94c8f81ad198ebc4df6ae: Log matching property violated. Preceding OpId in replica: term: 6 index: 34. Preceding OpId from leader: term: 7 index: 35. (index mismatch)
I20251026 08:53:08.665970 4837 consensus_queue.cc:1048] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 34, Time since last communication: 0.000s
I20251026 08:53:09.688758 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:09.691412 3907 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:47610
I20251026 08:53:09.691622 3907 raft_consensus.cc:606] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:53:09.692149 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:09.692364 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:09.964207 4843 raft_consensus.cc:993] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:09.964409 4843 raft_consensus.cc:1081] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:09.965055 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60842
I20251026 08:53:09.965246 3835 raft_consensus.cc:493] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:09.965350 3835 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:09.967234 3835 raft_consensus.cc:515] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:09.967782 3835 leader_election.cc:290] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 5 election: Requested vote from peers 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67:33171), 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:53:09.968134 3907 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "53f35e5629a94c8f81ad198ebc4df6ae"
I20251026 08:53:09.968303 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b1edddc92bb0463b9ed275da3c7154ef" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
I20251026 08:53:09.968339 3907 raft_consensus.cc:3055] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Stepping down as leader of term 4
I20251026 08:53:09.968465 4226 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:09.968500 3907 raft_consensus.cc:740] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 53f35e5629a94c8f81ad198ebc4df6ae, State: Running, Role: LEADER
I20251026 08:53:09.968915 3907 consensus_queue.cc:260] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:09.969328 3907 raft_consensus.cc:3060] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:09.970592 4226 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 5.
I20251026 08:53:09.970947 3801 leader_election.cc:304] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9ac1b52f24abf8690af5a6924646c, 96cd069722b34c1eab8ae9a2f953dd7a; no voters:
I20251026 08:53:09.971184 4826 raft_consensus.cc:2804] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Leader election won for term 5
I20251026 08:53:09.971513 4826 raft_consensus.cc:697] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:09.971746 4826 consensus_queue.cc:237] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:09.972365 3907 raft_consensus.cc:2468] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 5.
I20251026 08:53:09.974356 3650 catalog_manager.cc:5649] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 4 to 5, leader changed from 53f35e5629a94c8f81ad198ebc4df6ae (127.0.103.67) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 5 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:10.401575 4226 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 5 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20251026 08:53:10.402098 4826 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:10.405503 3907 raft_consensus.cc:1275] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20251026 08:53:10.406605 4852 consensus_queue.cc:1048] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "53f35e5629a94c8f81ad198ebc4df6ae" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 33171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:11.692708 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:11.695043 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:53:11.695204 3835 raft_consensus.cc:606] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:53:11.695685 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:11.695842 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:12.168067 4859 raft_consensus.cc:993] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:12.168193 4859 raft_consensus.cc:1081] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:12.168696 3754 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51952
I20251026 08:53:12.168859 3754 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:12.168959 3754 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:12.170873 3754 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:12.171416 3754 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 3 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:53:12.171710 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:12.171764 4300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "b983c9a7c2924030a396232a4ad46572"
I20251026 08:53:12.171923 3835 raft_consensus.cc:3055] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:53:12.171936 4300 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:12.172046 3835 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:12.172258 3835 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:12.172513 3835 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:12.173705 4300 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 3.
I20251026 08:53:12.174003 3724 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b983c9a7c2924030a396232a4ad46572, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:12.174147 3835 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 3.
I20251026 08:53:12.174264 4861 raft_consensus.cc:2804] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:53:12.174392 4861 raft_consensus.cc:697] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:12.174609 4861 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:12.176440 3656 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 2 to 3, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:12.661876 3835 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:53:12.662475 4861 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:12.666160 4300 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 2 index: 30. Preceding OpId from leader: term: 3 index: 31. (index mismatch)
I20251026 08:53:12.666742 4861 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 31, Last known committed idx: 30, Time since last communication: 0.000s
I20251026 08:53:13.696154 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:13.698522 3754 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:53:13.698691 3754 raft_consensus.cc:606] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:53:13.699123 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:13.699348 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:13.946266 4877 raft_consensus.cc:993] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:13.946455 4877 raft_consensus.cc:1081] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:13.947011 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60858
I20251026 08:53:13.947166 3835 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:13.947247 3835 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:53:13.949378 3835 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:13.950037 3835 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 4 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:53:13.950412 3754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:53:13.950466 4300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: true dest_uuid: "b983c9a7c2924030a396232a4ad46572"
I20251026 08:53:13.950652 3754 raft_consensus.cc:3055] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Stepping down as leader of term 3
I20251026 08:53:13.950650 4300 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:53:13.950775 3754 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:13.950995 3754 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:13.951256 3754 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Advancing to term 4
I20251026 08:53:13.952579 4300 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 4.
I20251026 08:53:13.952982 3799 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, b983c9a7c2924030a396232a4ad46572; no voters:
I20251026 08:53:13.953145 3754 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 4.
I20251026 08:53:13.953279 4859 raft_consensus.cc:2804] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Leader election won for term 4
I20251026 08:53:13.953658 4859 raft_consensus.cc:697] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:13.953908 4859 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 31, Committed index: 31, Last appended: 3.31, Last appended by leader: 31, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:13.956270 3650 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 3 to 4, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 4 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:14.358484 3754 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251026 08:53:14.359009 4859 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:14.363672 4300 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 4 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 3 index: 31. Preceding OpId from leader: term: 4 index: 32. (index mismatch)
I20251026 08:53:14.364163 4859 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 32, Last known committed idx: 31, Time since last communication: 0.000s
I20251026 08:53:15.699736 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:15.702549 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:53:15.702719 3835 raft_consensus.cc:606] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:53:15.703154 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:15.703329 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:15.978574 4889 raft_consensus.cc:993] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:15.978829 4890 raft_consensus.cc:1081] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:15.979368 3754 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51952
I20251026 08:53:15.979578 3754 raft_consensus.cc:493] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:15.979681 3754 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:15.981410 3754 raft_consensus.cc:515] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:15.981971 3754 leader_election.cc:290] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 5 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), b983c9a7c2924030a396232a4ad46572 (127.0.103.70:46541)
I20251026 08:53:15.982318 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:15.982407 4300 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd57692a7704c9690a892319e4bf446" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 5 candidate_status { last_received { term: 4 index: 32 } } ignore_live_leader: true dest_uuid: "b983c9a7c2924030a396232a4ad46572"
I20251026 08:53:15.982491 3835 raft_consensus.cc:3055] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Stepping down as leader of term 4
I20251026 08:53:15.982568 3835 raft_consensus.cc:740] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:15.982606 4300 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:15.982803 3835 consensus_queue.cc:260] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:15.983070 3835 raft_consensus.cc:3060] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 4 FOLLOWER]: Advancing to term 5
I20251026 08:53:15.984962 4300 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 5.
I20251026 08:53:15.984982 3835 raft_consensus.cc:2468] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 5.
I20251026 08:53:15.985370 3726 leader_election.cc:304] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, b983c9a7c2924030a396232a4ad46572, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:15.985635 4877 raft_consensus.cc:2804] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Leader election won for term 5
I20251026 08:53:15.985932 4877 raft_consensus.cc:697] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:15.986160 4877 consensus_queue.cc:237] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 32, Committed index: 32, Last appended: 4.32, Last appended by leader: 32, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } }
I20251026 08:53:15.987959 3656 catalog_manager.cc:5649] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 4 to 5, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 5 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 30 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:16.400134 4300 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20251026 08:53:16.400692 4877 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b983c9a7c2924030a396232a4ad46572" member_type: VOTER last_known_addr { host: "127.0.103.70" port: 46541 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:16.404040 3835 raft_consensus.cc:1275] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Refusing update from remote peer e7be295d3ba247a1a4475fa020d1ade1: Log matching property violated. Preceding OpId in replica: term: 4 index: 32. Preceding OpId from leader: term: 5 index: 33. (index mismatch)
I20251026 08:53:16.404578 4891 consensus_queue.cc:1048] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 33, Last known committed idx: 32, Time since last communication: 0.000s
I20251026 08:53:17.703644 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:17.705868 3754 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
mode: GRACEFUL
new_leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:46322
I20251026 08:53:17.706023 3754 raft_consensus.cc:606] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Received request to transfer leadership to TS 96cd069722b34c1eab8ae9a2f953dd7a
I20251026 08:53:17.706421 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:17.706604 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:17.849217 4902 raft_consensus.cc:993] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: : Instructing follower 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:17.849351 4902 raft_consensus.cc:1081] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Signalling peer 96cd069722b34c1eab8ae9a2f953dd7a to start an election
I20251026 08:53:17.849886 3835 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
from {username='slave'} at 127.0.0.1:60858
I20251026 08:53:17.850083 3835 raft_consensus.cc:493] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:17.850183 3835 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:17.851931 3835 raft_consensus.cc:515] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:17.852639 3835 leader_election.cc:290] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 2 election: Requested vote from peers e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515), 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:53:17.852952 3754 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
I20251026 08:53:17.853034 4226 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: true dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
I20251026 08:53:17.853116 3754 raft_consensus.cc:3055] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:53:17.853176 3754 raft_consensus.cc:740] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:17.853216 4226 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:17.853384 3754 consensus_queue.cc:260] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:17.853641 3754 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:17.855197 3754 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 2.
I20251026 08:53:17.855484 4226 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96cd069722b34c1eab8ae9a2f953dd7a in term 2.
I20251026 08:53:17.855504 3802 leader_election.cc:304] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96cd069722b34c1eab8ae9a2f953dd7a, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:17.855813 4890 raft_consensus.cc:2804] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:53:17.856206 4890 raft_consensus.cc:697] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Becoming Leader. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:17.856480 4890 consensus_queue.cc:237] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 29, Committed index: 29, Last appended: 1.29, Last appended by leader: 29, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:17.858549 3650 catalog_manager.cc:5649] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a reported cstate change: term changed from 1 to 2, leader changed from e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65) to 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66). New cstate: current_term: 2 leader_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:18.246729 4226 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:53:18.247220 4890 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:18.250885 3754 raft_consensus.cc:1275] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Refusing update from remote peer 96cd069722b34c1eab8ae9a2f953dd7a: Log matching property violated. Preceding OpId in replica: term: 1 index: 29. Preceding OpId from leader: term: 2 index: 30. (index mismatch)
I20251026 08:53:18.251576 4904 consensus_queue.cc:1048] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 30, Last known committed idx: 29, Time since last communication: 0.000s
I20251026 08:53:19.707031 3712 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:19.709445 3835 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760"
dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
mode: GRACEFUL
new_leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:60876
I20251026 08:53:19.709599 3835 raft_consensus.cc:606] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Received request to transfer leadership to TS e7be295d3ba247a1a4475fa020d1ade1
I20251026 08:53:19.709975 3712 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20251026 08:53:19.710144 3712 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:19.853826 4917 raft_consensus.cc:993] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: : Instructing follower e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:19.853950 4917 raft_consensus.cc:1081] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Signalling peer e7be295d3ba247a1a4475fa020d1ade1 to start an election
I20251026 08:53:19.854451 3754 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760"
dest_uuid: "e7be295d3ba247a1a4475fa020d1ade1"
from {username='slave'} at 127.0.0.1:51952
I20251026 08:53:19.854594 3754 raft_consensus.cc:493] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:19.854658 3754 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:19.856365 3754 raft_consensus.cc:515] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:19.856904 3754 leader_election.cc:290] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 3 election: Requested vote from peers 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66:37913), 49e9ac1b52f24abf8690af5a6924646c (127.0.103.69:34723)
I20251026 08:53:19.857234 3835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "96cd069722b34c1eab8ae9a2f953dd7a"
I20251026 08:53:19.857357 4224 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8c430afe1db8423b97c3c436cb912760" candidate_uuid: "e7be295d3ba247a1a4475fa020d1ade1" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: true dest_uuid: "49e9ac1b52f24abf8690af5a6924646c"
I20251026 08:53:19.857371 3835 raft_consensus.cc:3055] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:53:19.857482 3835 raft_consensus.cc:740] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96cd069722b34c1eab8ae9a2f953dd7a, State: Running, Role: LEADER
I20251026 08:53:19.857506 4224 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:19.857673 3835 consensus_queue.cc:260] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:19.857900 3835 raft_consensus.cc:3060] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:53:19.859414 4224 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 3.
I20251026 08:53:19.859819 3726 leader_election.cc:304] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49e9ac1b52f24abf8690af5a6924646c, e7be295d3ba247a1a4475fa020d1ade1; no voters:
I20251026 08:53:19.859895 3835 raft_consensus.cc:2468] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7be295d3ba247a1a4475fa020d1ade1 in term 3.
I20251026 08:53:19.860071 4902 raft_consensus.cc:2804] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:53:19.860528 4902 raft_consensus.cc:697] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Becoming Leader. State: Replica: e7be295d3ba247a1a4475fa020d1ade1, State: Running, Role: LEADER
I20251026 08:53:19.860816 4902 consensus_queue.cc:237] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 2.30, Last appended by leader: 30, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } }
I20251026 08:53:19.863176 3656 catalog_manager.cc:5649] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 reported cstate change: term changed from 2 to 3, leader changed from 96cd069722b34c1eab8ae9a2f953dd7a (127.0.103.66) to e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "e7be295d3ba247a1a4475fa020d1ade1" committed_config { opid_index: 29 OBSOLETE_local: false peers { permanent_uuid: "e7be295d3ba247a1a4475fa020d1ade1" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 39515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96cd069722b34c1eab8ae9a2f953dd7a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49e9ac1b52f24abf8690af5a6924646c" member_type: VOTER last_known_addr { host: "127.0.103.69" port: 34723 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:195: Failure
Expected equality of these values:
0
NumMovesScheduled(leader_idx, BalanceThreadType::LEADER_REBALANCE)
Which is: 1
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:403: Failure
Failed
Timed out waiting for assertion to pass.
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:575: Failure
Expected: CheckNoLeaderMovesScheduled() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20251026 08:53:19.932608 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:53:19.944420 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:19.944651 413 tablet_replica.cc:333] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:19.944844 413 raft_consensus.cc:2243] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 LEADER]: Raft consensus shutting down.
I20251026 08:53:19.945114 413 raft_consensus.cc:2272] T 6dd57692a7704c9690a892319e4bf446 P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.945652 413 tablet_replica.cc:333] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:19.945787 413 raft_consensus.cc:2243] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.945959 413 raft_consensus.cc:2272] T 85fe4797e56f4170876f76414e23628f P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.946419 413 tablet_replica.cc:333] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:19.946532 413 raft_consensus.cc:2243] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 5 LEADER]: Raft consensus shutting down.
I20251026 08:53:19.946727 413 raft_consensus.cc:2272] T 3d3135b8b6fe4dff81329c307264ea9f P e7be295d3ba247a1a4475fa020d1ade1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.947139 413 tablet_replica.cc:333] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:19.947234 413 raft_consensus.cc:2243] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.947395 413 raft_consensus.cc:2272] T 0409b48759d34a6fa9c86d88b809e093 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.947788 413 tablet_replica.cc:333] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:19.947907 413 raft_consensus.cc:2243] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:53:19.948072 413 pending_rounds.cc:70] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Trying to abort 1 pending ops.
I20251026 08:53:19.948112 413 pending_rounds.cc:77] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1: Aborting op as it isn't in flight: id { term: 3 index: 31 } timestamp: 7214976204232638464 op_type: NO_OP noop_request { }
I20251026 08:53:19.948206 413 raft_consensus.cc:2889] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:19.948288 413 raft_consensus.cc:2272] T 8c430afe1db8423b97c3c436cb912760 P e7be295d3ba247a1a4475fa020d1ade1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.948778 413 tablet_replica.cc:333] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1: stopping tablet replica
I20251026 08:53:19.948887 413 raft_consensus.cc:2243] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.949024 413 raft_consensus.cc:2272] T c5f200f20b1741b4a2c6bc379b2ae8f5 P e7be295d3ba247a1a4475fa020d1ade1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.964805 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:53:19.969647 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:53:19.994081 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:19.994303 413 tablet_replica.cc:333] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:19.994444 413 raft_consensus.cc:2243] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 LEADER]: Raft consensus shutting down.
I20251026 08:53:19.994660 413 raft_consensus.cc:2272] T b1edddc92bb0463b9ed275da3c7154ef P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.995179 413 tablet_replica.cc:333] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:19.995299 413 raft_consensus.cc:2243] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.995406 413 raft_consensus.cc:2272] T 6dd57692a7704c9690a892319e4bf446 P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.995790 413 tablet_replica.cc:333] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:19.995901 413 raft_consensus.cc:2243] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:53:19.996088 413 raft_consensus.cc:2272] T 5f3c583b37d74efe9630e3723a2c313e P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.996451 413 tablet_replica.cc:333] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:19.996553 413 raft_consensus.cc:2243] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.996652 413 raft_consensus.cc:2272] T 8c430afe1db8423b97c3c436cb912760 P 96cd069722b34c1eab8ae9a2f953dd7a [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.997025 413 tablet_replica.cc:333] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:19.997148 413 raft_consensus.cc:2243] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.997258 413 raft_consensus.cc:2272] T 3d3135b8b6fe4dff81329c307264ea9f P 96cd069722b34c1eab8ae9a2f953dd7a [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:19.997658 413 tablet_replica.cc:333] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a: stopping tablet replica
I20251026 08:53:19.997789 413 raft_consensus.cc:2243] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 7 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:19.997910 413 raft_consensus.cc:2272] T 9ee117517b8b4ae18fa61970868914a4 P 96cd069722b34c1eab8ae9a2f953dd7a [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.013214 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
W20251026 08:53:20.014369 4192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39515: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20251026 08:53:20.015422 4192 consensus_peers.cc:597] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c -> Peer e7be295d3ba247a1a4475fa020d1ade1 (127.0.103.65:39515): Couldn't send request to peer e7be295d3ba247a1a4475fa020d1ade1. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:39515: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:20.018314 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:53:20.027504 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:20.027731 413 tablet_replica.cc:333] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:20.027899 413 raft_consensus.cc:2243] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.028031 413 raft_consensus.cc:2272] T b1edddc92bb0463b9ed275da3c7154ef P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.028496 413 tablet_replica.cc:333] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:20.028620 413 raft_consensus.cc:2243] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 7 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.028869 413 raft_consensus.cc:2272] T 9ee117517b8b4ae18fa61970868914a4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.029279 413 tablet_replica.cc:333] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:20.029399 413 raft_consensus.cc:2243] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.029597 413 raft_consensus.cc:2272] T c1e736aea2b54b378ffbe821cb9e42c4 P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.030001 413 tablet_replica.cc:333] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:20.030143 413 raft_consensus.cc:2243] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.030259 413 raft_consensus.cc:2272] T 3d3135b8b6fe4dff81329c307264ea9f P 53f35e5629a94c8f81ad198ebc4df6ae [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.030642 413 tablet_replica.cc:333] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:20.030761 413 raft_consensus.cc:2243] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.030992 413 raft_consensus.cc:2272] T d0a49b69e0a94ae9827e91c43006c2d5 P 53f35e5629a94c8f81ad198ebc4df6ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.031366 413 tablet_replica.cc:333] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae: stopping tablet replica
I20251026 08:53:20.031476 413 raft_consensus.cc:2243] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.031656 413 raft_consensus.cc:2272] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 53f35e5629a94c8f81ad198ebc4df6ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.047941 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:53:20.067610 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:53:20.075846 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:20.076051 413 tablet_replica.cc:333] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739: stopping tablet replica
I20251026 08:53:20.076223 413 raft_consensus.cc:2243] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.076431 413 raft_consensus.cc:2272] T 85fe4797e56f4170876f76414e23628f P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.076928 413 tablet_replica.cc:333] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739: stopping tablet replica
I20251026 08:53:20.077046 413 raft_consensus.cc:2243] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.077206 413 raft_consensus.cc:2272] T c1e736aea2b54b378ffbe821cb9e42c4 P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.077584 413 tablet_replica.cc:333] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739: stopping tablet replica
I20251026 08:53:20.077708 413 raft_consensus.cc:2243] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.077831 413 raft_consensus.cc:2272] T 5f3c583b37d74efe9630e3723a2c313e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.078231 413 tablet_replica.cc:333] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739: stopping tablet replica
I20251026 08:53:20.078341 413 raft_consensus.cc:2243] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.078508 413 raft_consensus.cc:2272] T 0409b48759d34a6fa9c86d88b809e093 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.078924 413 tablet_replica.cc:333] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739: stopping tablet replica
I20251026 08:53:20.079044 413 raft_consensus.cc:2243] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.079278 413 raft_consensus.cc:2272] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 381d5756da8c4fa3ab5663d92bf76739 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.079774 413 tablet_replica.cc:333] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739: stopping tablet replica
I20251026 08:53:20.079908 413 raft_consensus.cc:2243] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.080149 413 raft_consensus.cc:2272] T d0a49b69e0a94ae9827e91c43006c2d5 P 381d5756da8c4fa3ab5663d92bf76739 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.094326 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:53:20.097524 413 tablet_server.cc:178] TabletServer@127.0.103.69:0 shutting down...
I20251026 08:53:20.105974 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:20.106208 413 tablet_replica.cc:333] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c: stopping tablet replica
I20251026 08:53:20.106339 413 raft_consensus.cc:2243] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.106516 413 raft_consensus.cc:2272] T 0b2c4bd03bfd4efd8aa306d8557fad3e P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.106913 413 tablet_replica.cc:333] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c: stopping tablet replica
I20251026 08:53:20.107025 413 raft_consensus.cc:2243] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.107167 413 raft_consensus.cc:2272] T c5f200f20b1741b4a2c6bc379b2ae8f5 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.107524 413 tablet_replica.cc:333] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c: stopping tablet replica
I20251026 08:53:20.107630 413 raft_consensus.cc:2243] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.107731 413 raft_consensus.cc:2272] T b1edddc92bb0463b9ed275da3c7154ef P 49e9ac1b52f24abf8690af5a6924646c [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.108088 413 tablet_replica.cc:333] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c: stopping tablet replica
I20251026 08:53:20.108192 413 raft_consensus.cc:2243] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.108328 413 raft_consensus.cc:2272] T 8c430afe1db8423b97c3c436cb912760 P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.108690 413 tablet_replica.cc:333] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c: stopping tablet replica
I20251026 08:53:20.108791 413 raft_consensus.cc:2243] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.109015 413 raft_consensus.cc:2272] T 85fe4797e56f4170876f76414e23628f P 49e9ac1b52f24abf8690af5a6924646c [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.109380 413 tablet_replica.cc:333] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c: stopping tablet replica
I20251026 08:53:20.109478 413 raft_consensus.cc:2243] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.109670 413 raft_consensus.cc:2272] T 0409b48759d34a6fa9c86d88b809e093 P 49e9ac1b52f24abf8690af5a6924646c [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.123834 413 tablet_server.cc:195] TabletServer@127.0.103.69:0 shutdown complete.
I20251026 08:53:20.126957 413 tablet_server.cc:178] TabletServer@127.0.103.70:0 shutting down...
I20251026 08:53:20.135032 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:20.135215 413 tablet_replica.cc:333] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572: stopping tablet replica
I20251026 08:53:20.135346 413 raft_consensus.cc:2243] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.135552 413 raft_consensus.cc:2272] T 5f3c583b37d74efe9630e3723a2c313e P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.135944 413 tablet_replica.cc:333] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572: stopping tablet replica
I20251026 08:53:20.136054 413 raft_consensus.cc:2243] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 7 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.136162 413 raft_consensus.cc:2272] T 9ee117517b8b4ae18fa61970868914a4 P b983c9a7c2924030a396232a4ad46572 [term 7 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.136500 413 tablet_replica.cc:333] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572: stopping tablet replica
I20251026 08:53:20.136605 413 raft_consensus.cc:2243] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.136739 413 raft_consensus.cc:2272] T d0a49b69e0a94ae9827e91c43006c2d5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.137077 413 tablet_replica.cc:333] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572: stopping tablet replica
I20251026 08:53:20.137177 413 raft_consensus.cc:2243] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.137367 413 raft_consensus.cc:2272] T c5f200f20b1741b4a2c6bc379b2ae8f5 P b983c9a7c2924030a396232a4ad46572 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.137691 413 tablet_replica.cc:333] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572: stopping tablet replica
I20251026 08:53:20.137794 413 raft_consensus.cc:2243] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:20.137900 413 raft_consensus.cc:2272] T 6dd57692a7704c9690a892319e4bf446 P b983c9a7c2924030a396232a4ad46572 [term 5 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.138312 413 tablet_replica.cc:333] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572: stopping tablet replica
I20251026 08:53:20.138419 413 raft_consensus.cc:2243] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.138636 413 raft_consensus.cc:2272] T c1e736aea2b54b378ffbe821cb9e42c4 P b983c9a7c2924030a396232a4ad46572 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.166698 413 tablet_server.cc:195] TabletServer@127.0.103.70:0 shutdown complete.
I20251026 08:53:20.170267 413 master.cc:561] Master@127.0.103.126:39299 shutting down...
I20251026 08:53:20.177305 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:20.177464 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:20.177541 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 69d6bf04204a4fafb6d6c224711cc3c8: stopping tablet replica
I20251026 08:53:20.180989 413 master.cc:583] Master@127.0.103.126:39299 shutdown complete.
I20251026 08:53:20.186820 413 test_util.cc:183] -----------------------------------------------
I20251026 08:53:20.186879 413 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestMaxMovesPerServer.1761468669168814-413-0
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer (37120 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20251026 08:53:20.188409 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:35743
I20251026 08:53:20.188791 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:20.190968 4929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:20.190979 4928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:20.191268 413 server_base.cc:1047] running on GCE node
W20251026 08:53:20.191349 4931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:20.191603 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:20.191658 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:20.191706 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468800191703 us; error 0 us; skew 500 ppm
I20251026 08:53:20.191830 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:20.192755 413 webserver.cc:492] Webserver started at http://127.0.103.126:36223/ using document root <none> and password file <none>
I20251026 08:53:20.192925 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:20.192986 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:20.193101 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:20.193712 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "9f7113ff12024fadbbde37744a622d87"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.195776 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:20.196919 4936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.197206 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:20.197274 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "9f7113ff12024fadbbde37744a622d87"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.197360 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:20.204675 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:20.205049 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:20.214411 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:35743
I20251026 08:53:20.214430 4987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:35743 every 8 connection(s)
I20251026 08:53:20.215404 4988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:20.219066 4988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87: Bootstrap starting.
I20251026 08:53:20.220113 4988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:20.221365 4988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87: No bootstrap required, opened a new log
I20251026 08:53:20.221791 4988 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7113ff12024fadbbde37744a622d87" member_type: VOTER }
I20251026 08:53:20.221886 4988 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:20.221920 4988 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f7113ff12024fadbbde37744a622d87, State: Initialized, Role: FOLLOWER
I20251026 08:53:20.222083 4988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7113ff12024fadbbde37744a622d87" member_type: VOTER }
I20251026 08:53:20.222177 4988 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:53:20.222210 4988 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:53:20.222254 4988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:20.223595 4988 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7113ff12024fadbbde37744a622d87" member_type: VOTER }
I20251026 08:53:20.223724 4988 leader_election.cc:304] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f7113ff12024fadbbde37744a622d87; no voters:
I20251026 08:53:20.223980 4988 leader_election.cc:290] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:53:20.224082 4991 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:20.224450 4991 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 1 LEADER]: Becoming Leader. State: Replica: 9f7113ff12024fadbbde37744a622d87, State: Running, Role: LEADER
I20251026 08:53:20.224599 4988 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:53:20.224609 4991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7113ff12024fadbbde37744a622d87" member_type: VOTER }
I20251026 08:53:20.225235 4992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f7113ff12024fadbbde37744a622d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7113ff12024fadbbde37744a622d87" member_type: VOTER } }
I20251026 08:53:20.225288 4993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f7113ff12024fadbbde37744a622d87. Latest consensus state: current_term: 1 leader_uuid: "9f7113ff12024fadbbde37744a622d87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f7113ff12024fadbbde37744a622d87" member_type: VOTER } }
I20251026 08:53:20.225395 4992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:20.225427 4993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:20.225838 4996 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:53:20.227051 4996 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:53:20.228825 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:53:20.229380 4996 catalog_manager.cc:1357] Generated new cluster ID: f2dcc7a8894544b2a66b1a597a6c85f3
I20251026 08:53:20.229444 4996 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:53:20.255573 4996 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:53:20.256165 4996 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:53:20.261031 4996 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87: Generated new TSK 0
I20251026 08:53:20.261193 4996 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:53:20.293509 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:20.295789 5009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:20.295957 5010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:20.297534 5012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:20.297624 413 server_base.cc:1047] running on GCE node
I20251026 08:53:20.301585 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:20.301636 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:20.301656 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468800301653 us; error 0 us; skew 500 ppm
I20251026 08:53:20.301779 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:20.302736 413 webserver.cc:492] Webserver started at http://127.0.103.65:45939/ using document root <none> and password file <none>
I20251026 08:53:20.302924 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:20.302991 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:20.303110 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:20.303613 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "6bd26e15a57f49c298b66adbf108b46b"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.305810 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:53:20.307161 5017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.307463 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:20.307569 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "6bd26e15a57f49c298b66adbf108b46b"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.307670 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:20.315522 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:20.315918 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:20.316344 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:20.317034 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:20.317108 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.317186 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:20.317235 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.328284 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:38761
I20251026 08:53:20.328308 5079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:38761 every 8 connection(s)
I20251026 08:53:20.329541 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:20.332060 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:20.332566 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:20.332648 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:20.333099 413 server_base.cc:1047] running on GCE node
I20251026 08:53:20.333295 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:20.333371 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:20.333429 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468800333407 us; error 0 us; skew 500 ppm
I20251026 08:53:20.333582 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:20.334692 413 webserver.cc:492] Webserver started at http://127.0.103.66:39381/ using document root <none> and password file <none>
I20251026 08:53:20.334941 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:20.335034 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:20.335150 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:20.335788 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "4095c3e315d74b57b085b807cc11f5c5"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.336202 5080 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35743
I20251026 08:53:20.336318 5080 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:20.336598 5080 heartbeater.cc:507] Master 127.0.103.126:35743 requested a full tablet report, sending...
I20251026 08:53:20.337301 4953 ts_manager.cc:194] Registered new tserver with Master: 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761)
I20251026 08:53:20.337939 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:20.338336 4953 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53400
I20251026 08:53:20.339464 5092 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.339781 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:20.339884 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "4095c3e315d74b57b085b807cc11f5c5"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.339983 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:20.345151 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:20.345582 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:20.346030 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:20.346698 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:20.346778 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.346868 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:20.346927 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.358458 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:43541
I20251026 08:53:20.358489 5154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:43541 every 8 connection(s)
I20251026 08:53:20.359767 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:20.362175 5159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:20.362249 5158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:20.364531 5155 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35743
I20251026 08:53:20.364648 5155 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:20.364912 5155 heartbeater.cc:507] Master 127.0.103.126:35743 requested a full tablet report, sending...
I20251026 08:53:20.365551 4953 ts_manager.cc:194] Registered new tserver with Master: 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541)
I20251026 08:53:20.366158 413 server_base.cc:1047] running on GCE node
I20251026 08:53:20.366185 4953 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53414
W20251026 08:53:20.366443 5161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:20.366737 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:20.366827 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:20.366891 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468800366870 us; error 0 us; skew 500 ppm
I20251026 08:53:20.367027 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:20.367940 413 webserver.cc:492] Webserver started at http://127.0.103.67:40601/ using document root <none> and password file <none>
I20251026 08:53:20.368125 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:20.368191 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:20.368291 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:20.368803 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "99f502d0d17e487f9f856f9719b324c4"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.370962 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:20.372289 5166 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.372560 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:20.372663 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "99f502d0d17e487f9f856f9719b324c4"
format_stamp: "Formatted at 2025-10-26 08:53:20 on dist-test-slave-rd83"
I20251026 08:53:20.372784 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:20.384610 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:20.384969 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:20.385408 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:20.386094 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:20.386180 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.386288 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:20.386349 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:20.396912 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:35361
I20251026 08:53:20.396937 5228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:35361 every 8 connection(s)
I20251026 08:53:20.402092 5229 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35743
I20251026 08:53:20.402194 5229 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:20.402458 5229 heartbeater.cc:507] Master 127.0.103.126:35743 requested a full tablet report, sending...
I20251026 08:53:20.403077 4953 ts_manager.cc:194] Registered new tserver with Master: 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67:35361)
I20251026 08:53:20.403244 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005615438s
I20251026 08:53:20.403685 4953 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53428
I20251026 08:53:21.339602 5080 heartbeater.cc:499] Master 127.0.103.126:35743 was elected leader, sending a full tablet report...
I20251026 08:53:21.367254 5155 heartbeater.cc:499] Master 127.0.103.126:35743 was elected leader, sending a full tablet report...
I20251026 08:53:21.404702 5229 heartbeater.cc:499] Master 127.0.103.126:35743 was elected leader, sending a full tablet report...
I20251026 08:53:21.429774 413 test_util.cc:276] Using random seed: -365874963
I20251026 08:53:21.439898 4953 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53440:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:53:21.440500 4953 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:53:21.455832 5194 tablet_service.cc:1505] Processing CreateTablet for tablet f572d7dd7bf1426d88ab48eaf24fc6d6 (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 08:53:21.456281 5120 tablet_service.cc:1505] Processing CreateTablet for tablet f572d7dd7bf1426d88ab48eaf24fc6d6 (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 08:53:21.456393 5119 tablet_service.cc:1505] Processing CreateTablet for tablet ba836a02819540db96233ecb92595945 (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 08:53:21.456396 5194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f572d7dd7bf1426d88ab48eaf24fc6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.456645 5120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f572d7dd7bf1426d88ab48eaf24fc6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.456743 5193 tablet_service.cc:1505] Processing CreateTablet for tablet ba836a02819540db96233ecb92595945 (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 08:53:21.457125 5193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba836a02819540db96233ecb92595945. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.462132 5119 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba836a02819540db96233ecb92595945. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.462464 5192 tablet_service.cc:1505] Processing CreateTablet for tablet b2c4c7fbb70448c0bdcf52c45418ae1d (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 08:53:21.462874 5192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c4c7fbb70448c0bdcf52c45418ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.466516 5118 tablet_service.cc:1505] Processing CreateTablet for tablet b2c4c7fbb70448c0bdcf52c45418ae1d (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 08:53:21.466900 5118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c4c7fbb70448c0bdcf52c45418ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.487844 5250 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: Bootstrap starting.
I20251026 08:53:21.488301 5249 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: Bootstrap starting.
I20251026 08:53:21.489681 5249 tablet_bootstrap.cc:654] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.490374 5250 tablet_bootstrap.cc:654] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.493992 5250 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: No bootstrap required, opened a new log
I20251026 08:53:21.493990 5249 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: No bootstrap required, opened a new log
I20251026 08:53:21.494207 5250 ts_tablet_manager.cc:1403] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20251026 08:53:21.494207 5249 ts_tablet_manager.cc:1403] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20251026 08:53:21.494884 5249 raft_consensus.cc:359] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.494884 5250 raft_consensus.cc:359] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.495064 5249 raft_consensus.cc:385] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.495105 5250 raft_consensus.cc:385] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.495110 5249 raft_consensus.cc:740] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.495196 5250 raft_consensus.cc:740] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4095c3e315d74b57b085b807cc11f5c5, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.495306 5045 tablet_service.cc:1505] Processing CreateTablet for tablet f572d7dd7bf1426d88ab48eaf24fc6d6 (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION VALUES < 715827882
I20251026 08:53:21.495354 5249 consensus_queue.cc:260] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.495383 5250 consensus_queue.cc:260] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.495496 5044 tablet_service.cc:1505] Processing CreateTablet for tablet ba836a02819540db96233ecb92595945 (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20251026 08:53:21.495656 5045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f572d7dd7bf1426d88ab48eaf24fc6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.495836 5044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba836a02819540db96233ecb92595945. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.498446 5043 tablet_service.cc:1505] Processing CreateTablet for tablet b2c4c7fbb70448c0bdcf52c45418ae1d (DEFAULT_TABLE table=test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20251026 08:53:21.498857 5043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c4c7fbb70448c0bdcf52c45418ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:21.502101 5250 ts_tablet_manager.cc:1434] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: Time spent starting tablet: real 0.008s user 0.000s sys 0.007s
I20251026 08:53:21.505826 5250 tablet_bootstrap.cc:492] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5: Bootstrap starting.
I20251026 08:53:21.507076 5250 tablet_bootstrap.cc:654] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.508827 5249 ts_tablet_manager.cc:1434] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: Time spent starting tablet: real 0.015s user 0.003s sys 0.011s
I20251026 08:53:21.509135 5255 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b: Bootstrap starting.
I20251026 08:53:21.509435 5253 raft_consensus.cc:493] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:21.509558 5253 raft_consensus.cc:515] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.510273 5249 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: Bootstrap starting.
I20251026 08:53:21.510836 5255 tablet_bootstrap.cc:654] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.512210 5249 tablet_bootstrap.cc:654] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.518543 5250 tablet_bootstrap.cc:492] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5: No bootstrap required, opened a new log
I20251026 08:53:21.518692 5250 ts_tablet_manager.cc:1403] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20251026 08:53:21.519556 5250 raft_consensus.cc:359] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.519798 5250 raft_consensus.cc:385] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.519876 5250 raft_consensus.cc:740] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4095c3e315d74b57b085b807cc11f5c5, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.520164 5250 consensus_queue.cc:260] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.521348 5253 leader_election.cc:290] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541), 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761)
I20251026 08:53:21.523034 5250 ts_tablet_manager.cc:1434] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251026 08:53:21.525280 5250 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: Bootstrap starting.
I20251026 08:53:21.529371 5250 tablet_bootstrap.cc:654] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.537626 5130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f572d7dd7bf1426d88ab48eaf24fc6d6" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4095c3e315d74b57b085b807cc11f5c5" is_pre_election: true
I20251026 08:53:21.537885 5130 raft_consensus.cc:2468] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 0.
I20251026 08:53:21.538458 5168 leader_election.cc:304] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4095c3e315d74b57b085b807cc11f5c5, 99f502d0d17e487f9f856f9719b324c4; no voters:
I20251026 08:53:21.538898 5253 raft_consensus.cc:2804] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:21.539026 5253 raft_consensus.cc:493] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:21.539120 5253 raft_consensus.cc:3060] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.539837 5255 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b: No bootstrap required, opened a new log
I20251026 08:53:21.539970 5255 ts_tablet_manager.cc:1403] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.026s
I20251026 08:53:21.540342 5249 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: No bootstrap required, opened a new log
I20251026 08:53:21.540441 5250 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: No bootstrap required, opened a new log
I20251026 08:53:21.540539 5249 ts_tablet_manager.cc:1403] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.017s
I20251026 08:53:21.540660 5255 raft_consensus.cc:359] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.540840 5255 raft_consensus.cc:385] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.540899 5255 raft_consensus.cc:740] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd26e15a57f49c298b66adbf108b46b, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.540555 5250 ts_tablet_manager.cc:1403] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.002s
I20251026 08:53:21.541168 5255 consensus_queue.cc:260] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.541561 5249 raft_consensus.cc:359] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.541659 5253 raft_consensus.cc:515] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.541762 5249 raft_consensus.cc:385] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.541929 5249 raft_consensus.cc:740] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.541821 5250 raft_consensus.cc:359] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.542112 5250 raft_consensus.cc:385] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.542193 5250 raft_consensus.cc:740] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4095c3e315d74b57b085b807cc11f5c5, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.542332 5249 consensus_queue.cc:260] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.542397 5250 consensus_queue.cc:260] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.542719 5253 leader_election.cc:290] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 election: Requested vote from peers 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541), 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761)
I20251026 08:53:21.542943 5250 ts_tablet_manager.cc:1434] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:53:21.543397 5130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f572d7dd7bf1426d88ab48eaf24fc6d6" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4095c3e315d74b57b085b807cc11f5c5"
I20251026 08:53:21.543532 5130 raft_consensus.cc:3060] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.544049 5054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f572d7dd7bf1426d88ab48eaf24fc6d6" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd26e15a57f49c298b66adbf108b46b"
I20251026 08:53:21.544178 5054 raft_consensus.cc:3060] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.545993 5055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f572d7dd7bf1426d88ab48eaf24fc6d6" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd26e15a57f49c298b66adbf108b46b" is_pre_election: true
I20251026 08:53:21.546556 5249 ts_tablet_manager.cc:1434] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251026 08:53:21.547099 5054 raft_consensus.cc:2468] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 1.
I20251026 08:53:21.547269 5249 tablet_bootstrap.cc:492] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: Bootstrap starting.
I20251026 08:53:21.547384 5263 raft_consensus.cc:493] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:21.547521 5263 raft_consensus.cc:515] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.548352 5263 leader_election.cc:290] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761), 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541)
I20251026 08:53:21.548678 5129 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4095c3e315d74b57b085b807cc11f5c5" is_pre_election: true
I20251026 08:53:21.548887 5129 raft_consensus.cc:2468] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 0.
I20251026 08:53:21.549194 5249 tablet_bootstrap.cc:654] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.549242 5169 leader_election.cc:304] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd26e15a57f49c298b66adbf108b46b, 99f502d0d17e487f9f856f9719b324c4; no voters:
I20251026 08:53:21.549297 5168 leader_election.cc:304] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4095c3e315d74b57b085b807cc11f5c5, 99f502d0d17e487f9f856f9719b324c4; no voters:
I20251026 08:53:21.549561 5253 raft_consensus.cc:2804] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:21.549698 5130 raft_consensus.cc:2468] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 1.
I20251026 08:53:21.549952 5054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd26e15a57f49c298b66adbf108b46b" is_pre_election: true
W20251026 08:53:21.550447 5169 leader_election.cc:343] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761): Illegal state: must be running to vote when last-logged opid is not known
I20251026 08:53:21.550786 5263 raft_consensus.cc:2804] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:21.550876 5263 raft_consensus.cc:493] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:21.550948 5263 raft_consensus.cc:3060] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.552968 5255 ts_tablet_manager.cc:1434] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20251026 08:53:21.553272 5255 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Bootstrap starting.
I20251026 08:53:21.555267 5255 tablet_bootstrap.cc:654] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.561947 5253 raft_consensus.cc:697] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Becoming Leader. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Running, Role: LEADER
I20251026 08:53:21.562268 5253 consensus_queue.cc:237] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:21.563334 5255 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: No bootstrap required, opened a new log
I20251026 08:53:21.563444 5255 ts_tablet_manager.cc:1403] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.003s
I20251026 08:53:21.564301 5255 raft_consensus.cc:359] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.565788 5263 raft_consensus.cc:515] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.567220 5249 tablet_bootstrap.cc:492] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: No bootstrap required, opened a new log
I20251026 08:53:21.566818 5255 raft_consensus.cc:385] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.568133 5255 raft_consensus.cc:740] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd26e15a57f49c298b66adbf108b46b, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.568301 5249 ts_tablet_manager.cc:1403] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20251026 08:53:21.568375 5255 consensus_queue.cc:260] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.568915 5255 ts_tablet_manager.cc:1434] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251026 08:53:21.569206 5255 tablet_bootstrap.cc:492] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b: Bootstrap starting.
I20251026 08:53:21.569491 5249 raft_consensus.cc:359] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.569734 5249 raft_consensus.cc:385] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.569846 5249 raft_consensus.cc:740] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.567613 5054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd26e15a57f49c298b66adbf108b46b"
I20251026 08:53:21.570298 5054 raft_consensus.cc:3060] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.570329 5249 consensus_queue.cc:260] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.571125 5249 ts_tablet_manager.cc:1434] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251026 08:53:21.572503 5255 tablet_bootstrap.cc:654] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:21.572578 5054 raft_consensus.cc:2468] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 1.
I20251026 08:53:21.567613 5130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4095c3e315d74b57b085b807cc11f5c5"
I20251026 08:53:21.573024 5169 leader_election.cc:304] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd26e15a57f49c298b66adbf108b46b, 99f502d0d17e487f9f856f9719b324c4; no voters:
I20251026 08:53:21.573093 5130 raft_consensus.cc:3060] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.573318 5264 raft_consensus.cc:2804] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:21.575824 5130 raft_consensus.cc:2468] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 1.
I20251026 08:53:21.577657 4952 catalog_manager.cc:5649] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "99f502d0d17e487f9f856f9719b324c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:21.567284 5263 leader_election.cc:290] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761), 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541)
I20251026 08:53:21.579419 5264 raft_consensus.cc:697] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Becoming Leader. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Running, Role: LEADER
I20251026 08:53:21.579671 5264 consensus_queue.cc:237] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.580159 5255 tablet_bootstrap.cc:492] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b: No bootstrap required, opened a new log
I20251026 08:53:21.580325 5255 ts_tablet_manager.cc:1403] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
W20251026 08:53:21.580593 5081 tablet.cc:2378] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 08:53:21.581090 5255 raft_consensus.cc:359] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.581236 5255 raft_consensus.cc:385] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:21.581286 5255 raft_consensus.cc:740] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd26e15a57f49c298b66adbf108b46b, State: Initialized, Role: FOLLOWER
I20251026 08:53:21.581434 5255 consensus_queue.cc:260] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.581791 5255 ts_tablet_manager.cc:1434] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:53:21.583346 4951 catalog_manager.cc:5649] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "99f502d0d17e487f9f856f9719b324c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:21.589329 5264 raft_consensus.cc:493] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:21.589423 5264 raft_consensus.cc:515] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.589753 5264 leader_election.cc:290] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761), 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541)
I20251026 08:53:21.590062 5054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba836a02819540db96233ecb92595945" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd26e15a57f49c298b66adbf108b46b" is_pre_election: true
I20251026 08:53:21.590121 5130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba836a02819540db96233ecb92595945" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4095c3e315d74b57b085b807cc11f5c5" is_pre_election: true
I20251026 08:53:21.590209 5054 raft_consensus.cc:2468] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 0.
I20251026 08:53:21.590253 5130 raft_consensus.cc:2468] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 0.
I20251026 08:53:21.590493 5169 leader_election.cc:304] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd26e15a57f49c298b66adbf108b46b, 99f502d0d17e487f9f856f9719b324c4; no voters:
I20251026 08:53:21.590689 5264 raft_consensus.cc:2804] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:21.590751 5264 raft_consensus.cc:493] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:21.590796 5264 raft_consensus.cc:3060] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.592280 5264 raft_consensus.cc:515] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.592690 5264 leader_election.cc:290] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761), 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541)
I20251026 08:53:21.592908 5054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba836a02819540db96233ecb92595945" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd26e15a57f49c298b66adbf108b46b"
I20251026 08:53:21.593044 5054 raft_consensus.cc:3060] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.593021 5130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba836a02819540db96233ecb92595945" candidate_uuid: "99f502d0d17e487f9f856f9719b324c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4095c3e315d74b57b085b807cc11f5c5"
I20251026 08:53:21.593199 5130 raft_consensus.cc:3060] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:21.594619 5054 raft_consensus.cc:2468] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 1.
I20251026 08:53:21.594761 5130 raft_consensus.cc:2468] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f502d0d17e487f9f856f9719b324c4 in term 1.
I20251026 08:53:21.594913 5169 leader_election.cc:304] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd26e15a57f49c298b66adbf108b46b, 99f502d0d17e487f9f856f9719b324c4; no voters:
I20251026 08:53:21.595113 5264 raft_consensus.cc:2804] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:21.595239 5264 raft_consensus.cc:697] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Becoming Leader. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Running, Role: LEADER
I20251026 08:53:21.595427 5264 consensus_queue.cc:237] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:21.597064 4951 catalog_manager.cc:5649] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "99f502d0d17e487f9f856f9719b324c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:21.610694 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:21.612959 5272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:21.613085 5273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:21.613471 5275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:21.613670 413 server_base.cc:1047] running on GCE node
I20251026 08:53:21.613854 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:21.613906 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:21.613955 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468801613939 us; error 0 us; skew 500 ppm
I20251026 08:53:21.614140 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:21.615167 413 webserver.cc:492] Webserver started at http://127.0.103.68:37577/ using document root <none> and password file <none>
I20251026 08:53:21.615355 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:21.615415 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:21.615530 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:21.616014 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "dfa324dc29d844a58ef46c063cc6cd4e"
format_stamp: "Formatted at 2025-10-26 08:53:21 on dist-test-slave-rd83"
I20251026 08:53:21.617959 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:21.619295 5280 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:21.619561 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:21.619642 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "dfa324dc29d844a58ef46c063cc6cd4e"
format_stamp: "Formatted at 2025-10-26 08:53:21 on dist-test-slave-rd83"
I20251026 08:53:21.619791 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:21.631619 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:21.632010 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:21.632499 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:21.633486 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:21.633569 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:21.633669 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:21.633734 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:21.645776 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:34527
I20251026 08:53:21.645802 5342 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:34527 every 8 connection(s)
I20251026 08:53:21.648485 5264 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:53:21.651304 5269 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20251026 08:53:21.651384 5269 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20251026 08:53:21.654685 5343 heartbeater.cc:344] Connected to a master server at 127.0.103.126:35743
I20251026 08:53:21.654781 5343 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:21.655030 5343 heartbeater.cc:507] Master 127.0.103.126:35743 requested a full tablet report, sending...
I20251026 08:53:21.655592 4951 ts_manager.cc:194] Registered new tserver with Master: dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68:34527)
I20251026 08:53:21.656168 4951 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53444
I20251026 08:53:21.660998 5264 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:53:21.697232 5269 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:53:21.704444 5269 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:53:21.713932 5269 consensus_queue.cc:1048] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:53:21.717832 5269 consensus_queue.cc:1048] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:53:22.228775 5006 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:22.235580 5204 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d"
dest_uuid: "99f502d0d17e487f9f856f9719b324c4"
mode: GRACEFUL
new_leader_uuid: "4095c3e315d74b57b085b807cc11f5c5"
from {username='slave'} at 127.0.0.1:50066
I20251026 08:53:22.235868 5204 raft_consensus.cc:606] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Received request to transfer leadership to TS 4095c3e315d74b57b085b807cc11f5c5
W20251026 08:53:22.236565 5001 auto_rebalancer.cc:254] failed to send replica move request: Remote error: Service unavailable: leader transfer in progress
I20251026 08:53:22.236925 5203 tablet_service.cc:2066] Received LeaderStepDown RPC: tablet_id: "ba836a02819540db96233ecb92595945"
dest_uuid: "99f502d0d17e487f9f856f9719b324c4"
mode: GRACEFUL
new_leader_uuid: "6bd26e15a57f49c298b66adbf108b46b"
from {username='slave'} at 127.0.0.1:50066
I20251026 08:53:22.237087 5203 raft_consensus.cc:606] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Received request to transfer leadership to TS 6bd26e15a57f49c298b66adbf108b46b
I20251026 08:53:22.237385 5006 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20251026 08:53:22.237536 5006 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 08:53:22.237769 5001 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b2c4c7fbb70448c0bdcf52c45418ae1d, TS 99f502d0d17e487f9f856f9719b324c4 -> TS dfa324dc29d844a58ef46c063cc6cd4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
W20251026 08:53:22.237835 5001 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b2c4c7fbb70448c0bdcf52c45418ae1d, TS 99f502d0d17e487f9f856f9719b324c4 -> TS dfa324dc29d844a58ef46c063cc6cd4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
W20251026 08:53:22.238838 5001 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f572d7dd7bf1426d88ab48eaf24fc6d6, TS 6bd26e15a57f49c298b66adbf108b46b -> TS dfa324dc29d844a58ef46c063cc6cd4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
W20251026 08:53:22.238903 5001 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f572d7dd7bf1426d88ab48eaf24fc6d6, TS 6bd26e15a57f49c298b66adbf108b46b -> TS dfa324dc29d844a58ef46c063cc6cd4e move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } }
I20251026 08:53:22.371989 5253 raft_consensus.cc:993] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: : Instructing follower 4095c3e315d74b57b085b807cc11f5c5 to start an election
I20251026 08:53:22.372203 5253 raft_consensus.cc:1081] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Signalling peer 4095c3e315d74b57b085b807cc11f5c5 to start an election
I20251026 08:53:22.372783 5130 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d"
dest_uuid: "4095c3e315d74b57b085b807cc11f5c5"
from {username='slave'} at 127.0.0.1:60406
I20251026 08:53:22.372990 5130 raft_consensus.cc:493] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:22.373106 5130 raft_consensus.cc:3060] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:22.375065 5130 raft_consensus.cc:515] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:22.375701 5130 leader_election.cc:290] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [CANDIDATE]: Term 2 election: Requested vote from peers 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761), 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67:35361)
I20251026 08:53:22.381628 5054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d" candidate_uuid: "4095c3e315d74b57b085b807cc11f5c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6bd26e15a57f49c298b66adbf108b46b"
I20251026 08:53:22.381665 5204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2c4c7fbb70448c0bdcf52c45418ae1d" candidate_uuid: "4095c3e315d74b57b085b807cc11f5c5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99f502d0d17e487f9f856f9719b324c4"
I20251026 08:53:22.381790 5054 raft_consensus.cc:3060] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:22.381812 5204 raft_consensus.cc:3055] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:53:22.381906 5204 raft_consensus.cc:740] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Running, Role: LEADER
I20251026 08:53:22.382167 5204 consensus_queue.cc:260] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:22.382439 5204 raft_consensus.cc:3060] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:22.383603 5054 raft_consensus.cc:2468] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4095c3e315d74b57b085b807cc11f5c5 in term 2.
I20251026 08:53:22.383952 5095 leader_election.cc:304] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4095c3e315d74b57b085b807cc11f5c5, 6bd26e15a57f49c298b66adbf108b46b; no voters:
I20251026 08:53:22.384115 5204 raft_consensus.cc:2468] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4095c3e315d74b57b085b807cc11f5c5 in term 2.
I20251026 08:53:22.384501 5363 raft_consensus.cc:2804] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:53:22.384791 5363 raft_consensus.cc:697] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 LEADER]: Becoming Leader. State: Replica: 4095c3e315d74b57b085b807cc11f5c5, State: Running, Role: LEADER
I20251026 08:53:22.385072 5363 consensus_queue.cc:237] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:22.386731 4951 catalog_manager.cc:5649] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 reported cstate change: term changed from 1 to 2, leader changed from 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67) to 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "4095c3e315d74b57b085b807cc11f5c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:22.437358 5264 raft_consensus.cc:993] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: : Instructing follower 6bd26e15a57f49c298b66adbf108b46b to start an election
I20251026 08:53:22.437501 5264 raft_consensus.cc:1081] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Signalling peer 6bd26e15a57f49c298b66adbf108b46b to start an election
I20251026 08:53:22.438112 5054 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "ba836a02819540db96233ecb92595945"
dest_uuid: "6bd26e15a57f49c298b66adbf108b46b"
from {username='slave'} at 127.0.0.1:39588
I20251026 08:53:22.438294 5054 raft_consensus.cc:493] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20251026 08:53:22.438393 5054 raft_consensus.cc:3060] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:22.440151 5054 raft_consensus.cc:515] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:22.440829 5054 leader_election.cc:290] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [CANDIDATE]: Term 2 election: Requested vote from peers 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541), 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67:35361)
I20251026 08:53:22.447885 5204 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba836a02819540db96233ecb92595945" candidate_uuid: "6bd26e15a57f49c298b66adbf108b46b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99f502d0d17e487f9f856f9719b324c4"
I20251026 08:53:22.447885 5130 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba836a02819540db96233ecb92595945" candidate_uuid: "6bd26e15a57f49c298b66adbf108b46b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4095c3e315d74b57b085b807cc11f5c5"
I20251026 08:53:22.448115 5204 raft_consensus.cc:3055] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Stepping down as leader of term 1
I20251026 08:53:22.448113 5130 raft_consensus.cc:3060] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:22.448231 5204 raft_consensus.cc:740] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99f502d0d17e487f9f856f9719b324c4, State: Running, Role: LEADER
I20251026 08:53:22.448495 5204 consensus_queue.cc:260] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:22.448792 5204 raft_consensus.cc:3060] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:22.450573 5130 raft_consensus.cc:2468] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd26e15a57f49c298b66adbf108b46b in term 2.
I20251026 08:53:22.450954 5019 leader_election.cc:304] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4095c3e315d74b57b085b807cc11f5c5, 6bd26e15a57f49c298b66adbf108b46b; no voters:
I20251026 08:53:22.451079 5204 raft_consensus.cc:2468] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd26e15a57f49c298b66adbf108b46b in term 2.
I20251026 08:53:22.451514 5371 raft_consensus.cc:2804] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:53:22.451839 5371 raft_consensus.cc:697] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 2 LEADER]: Becoming Leader. State: Replica: 6bd26e15a57f49c298b66adbf108b46b, State: Running, Role: LEADER
I20251026 08:53:22.452061 5371 consensus_queue.cc:237] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } }
I20251026 08:53:22.453717 4951 catalog_manager.cc:5649] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b reported cstate change: term changed from 1 to 2, leader changed from 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67) to 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65). New cstate: current_term: 2 leader_uuid: "6bd26e15a57f49c298b66adbf108b46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:22.488569 5054 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:53:22.488963 5363 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:22.495616 5204 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:53:22.496026 5363 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:22.539923 5204 raft_consensus.cc:1275] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Refusing update from remote peer 6bd26e15a57f49c298b66adbf108b46b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:53:22.540316 5371 consensus_queue.cc:1048] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:22.570650 5130 raft_consensus.cc:1275] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Refusing update from remote peer 6bd26e15a57f49c298b66adbf108b46b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:53:22.571085 5371 consensus_queue.cc:1048] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:22.657065 5343 heartbeater.cc:499] Master 127.0.103.126:35743 was elected leader, sending a full tablet report...
I20251026 08:53:23.241756 5204 consensus_queue.cc:237] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } }
I20251026 08:53:23.243927 5130 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:53:23.244344 5054 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:53:23.244499 5264 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:23.245093 5264 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:23.256524 5129 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
I20251026 08:53:23.257920 5382 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
I20251026 08:53:23.261552 5054 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
I20251026 08:53:23.261901 4951 catalog_manager.cc:5649] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 reported cstate change: config changed from index -1 to 2, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "99f502d0d17e487f9f856f9719b324c4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
W20251026 08:53:23.266820 5170 consensus_peers.cc:597] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 -> Peer dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68:34527): Couldn't send request to peer dfa324dc29d844a58ef46c063cc6cd4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f572d7dd7bf1426d88ab48eaf24fc6d6. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:23.269742 5129 consensus_queue.cc:237] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } }
I20251026 08:53:23.271988 5054 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:23.272195 5204 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:23.272429 5390 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:23.272660 5389 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:23.274893 5204 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
I20251026 08:53:23.274941 5054 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
W20251026 08:53:23.280611 5096 consensus_peers.cc:597] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 -> Peer dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68:34527): Couldn't send request to peer dfa324dc29d844a58ef46c063cc6cd4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2c4c7fbb70448c0bdcf52c45418ae1d. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:23.320444 5364 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
I20251026 08:53:23.349545 5393 ts_tablet_manager.cc:933] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: Initiating tablet copy from peer 99f502d0d17e487f9f856f9719b324c4 (127.0.103.67:35361)
I20251026 08:53:23.350106 5393 tablet_copy_client.cc:323] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:35361
I20251026 08:53:23.353992 5397 ts_tablet_manager.cc:933] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: Initiating tablet copy from peer 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541)
I20251026 08:53:23.354672 5397 tablet_copy_client.cc:323] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:43541
I20251026 08:53:23.357523 5214 tablet_copy_service.cc:140] P 99f502d0d17e487f9f856f9719b324c4: Received BeginTabletCopySession request for tablet f572d7dd7bf1426d88ab48eaf24fc6d6 from peer dfa324dc29d844a58ef46c063cc6cd4e ({username='slave'} at 127.0.0.1:50128)
I20251026 08:53:23.357707 5214 tablet_copy_service.cc:161] P 99f502d0d17e487f9f856f9719b324c4: Beginning new tablet copy session on tablet f572d7dd7bf1426d88ab48eaf24fc6d6 from peer dfa324dc29d844a58ef46c063cc6cd4e at {username='slave'} at 127.0.0.1:50128: session id = dfa324dc29d844a58ef46c063cc6cd4e-f572d7dd7bf1426d88ab48eaf24fc6d6
I20251026 08:53:23.360232 5214 tablet_copy_source_session.cc:215] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:23.361460 5393 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f572d7dd7bf1426d88ab48eaf24fc6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:23.361425 5140 tablet_copy_service.cc:140] P 4095c3e315d74b57b085b807cc11f5c5: Received BeginTabletCopySession request for tablet b2c4c7fbb70448c0bdcf52c45418ae1d from peer dfa324dc29d844a58ef46c063cc6cd4e ({username='slave'} at 127.0.0.1:60438)
I20251026 08:53:23.361656 5140 tablet_copy_service.cc:161] P 4095c3e315d74b57b085b807cc11f5c5: Beginning new tablet copy session on tablet b2c4c7fbb70448c0bdcf52c45418ae1d from peer dfa324dc29d844a58ef46c063cc6cd4e at {username='slave'} at 127.0.0.1:60438: session id = dfa324dc29d844a58ef46c063cc6cd4e-b2c4c7fbb70448c0bdcf52c45418ae1d
I20251026 08:53:23.364741 5140 tablet_copy_source_session.cc:215] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:23.365554 4953 catalog_manager.cc:5649] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b reported cstate change: config changed from index -1 to 3, NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "4095c3e315d74b57b085b807cc11f5c5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } } }
I20251026 08:53:23.367187 5397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c4c7fbb70448c0bdcf52c45418ae1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:23.369439 5393 tablet_copy_client.cc:806] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:23.369654 5393 tablet_copy_client.cc:670] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:23.372277 5397 tablet_copy_client.cc:806] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:23.372543 5397 tablet_copy_client.cc:670] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:23.374503 5393 tablet_copy_client.cc:538] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:23.376755 5397 tablet_copy_client.cc:538] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:23.377557 5393 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: Bootstrap starting.
I20251026 08:53:23.379707 5397 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: Bootstrap starting.
I20251026 08:53:23.383890 5393 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:23.384325 5393 tablet_bootstrap.cc:492] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: Bootstrap complete.
I20251026 08:53:23.384568 5393 ts_tablet_manager.cc:1403] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20251026 08:53:23.385304 5393 raft_consensus.cc:359] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } }
I20251026 08:53:23.385519 5393 raft_consensus.cc:740] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dfa324dc29d844a58ef46c063cc6cd4e, State: Initialized, Role: LEARNER
I20251026 08:53:23.385723 5393 consensus_queue.cc:260] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } }
I20251026 08:53:23.385951 5397 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:23.386386 5397 tablet_bootstrap.cc:492] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: Bootstrap complete.
I20251026 08:53:23.386478 5393 ts_tablet_manager.cc:1434] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20251026 08:53:23.386619 5397 ts_tablet_manager.cc:1403] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: Time spent bootstrapping tablet: real 0.007s user 0.008s sys 0.000s
I20251026 08:53:23.387035 5214 tablet_copy_service.cc:342] P 99f502d0d17e487f9f856f9719b324c4: Request end of tablet copy session dfa324dc29d844a58ef46c063cc6cd4e-f572d7dd7bf1426d88ab48eaf24fc6d6 received from {username='slave'} at 127.0.0.1:50128
I20251026 08:53:23.387159 5214 tablet_copy_service.cc:434] P 99f502d0d17e487f9f856f9719b324c4: ending tablet copy session dfa324dc29d844a58ef46c063cc6cd4e-f572d7dd7bf1426d88ab48eaf24fc6d6 on tablet f572d7dd7bf1426d88ab48eaf24fc6d6 with peer dfa324dc29d844a58ef46c063cc6cd4e
I20251026 08:53:23.387995 5397 raft_consensus.cc:359] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } }
I20251026 08:53:23.388221 5397 raft_consensus.cc:740] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: dfa324dc29d844a58ef46c063cc6cd4e, State: Initialized, Role: LEARNER
I20251026 08:53:23.388442 5397 consensus_queue.cc:260] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: true } }
I20251026 08:53:23.388906 5397 ts_tablet_manager.cc:1434] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:53:23.389534 5140 tablet_copy_service.cc:342] P 4095c3e315d74b57b085b807cc11f5c5: Request end of tablet copy session dfa324dc29d844a58ef46c063cc6cd4e-b2c4c7fbb70448c0bdcf52c45418ae1d received from {username='slave'} at 127.0.0.1:60438
I20251026 08:53:23.389673 5140 tablet_copy_service.cc:434] P 4095c3e315d74b57b085b807cc11f5c5: ending tablet copy session dfa324dc29d844a58ef46c063cc6cd4e-b2c4c7fbb70448c0bdcf52c45418ae1d on tablet b2c4c7fbb70448c0bdcf52c45418ae1d with peer dfa324dc29d844a58ef46c063cc6cd4e
I20251026 08:53:23.441327 5318 raft_consensus.cc:1217] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251026 08:53:23.461629 5318 raft_consensus.cc:1217] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 08:53:23.566916 5392 raft_consensus.cc:1064] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: attempting to promote NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e to VOTER
I20251026 08:53:23.567519 5392 consensus_queue.cc:237] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } }
I20251026 08:53:23.569257 5318 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 LEARNER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:23.569322 5054 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:23.569742 5364 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:23.569981 5392 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:23.572031 5203 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:23.572532 5392 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:23.573535 5318 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.577860 5203 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.578001 4951 catalog_manager.cc:5649] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e reported cstate change: config changed from index 3 to 4, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "4095c3e315d74b57b085b807cc11f5c5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.580106 5389 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.583534 5054 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } attrs { replace: true } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.631749 5264 raft_consensus.cc:1064] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: attempting to promote NON_VOTER dfa324dc29d844a58ef46c063cc6cd4e to VOTER
I20251026 08:53:23.632560 5264 consensus_queue.cc:237] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } }
I20251026 08:53:23.634234 5318 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 LEARNER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:53:23.634234 5054 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:53:23.634651 5130 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:53:23.634688 5264 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:23.634940 5406 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:23.635126 5379 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:23.638027 5054 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.638561 5318 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.664182 4952 catalog_manager.cc:5649] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b reported cstate change: config changed from index 2 to 3, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "99f502d0d17e487f9f856f9719b324c4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.665207 5129 consensus_queue.cc:237] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } }
I20251026 08:53:23.666759 5318 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 08:53:23.667029 5203 raft_consensus.cc:1275] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Refusing update from remote peer 4095c3e315d74b57b085b807cc11f5c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251026 08:53:23.667389 5405 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:53:23.667630 5364 consensus_queue.cc:1048] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:53:23.670390 5318 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.670811 5203 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.674022 4950 catalog_manager.cc:5649] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 reported cstate change: config changed from index 4 to 5, VOTER 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "4095c3e315d74b57b085b807cc11f5c5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.716578 5379 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.720039 5129 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, dfa324dc29d844a58ef46c063cc6cd4e (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } attrs { replace: true } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.731485 5045 tablet_service.cc:1552] Processing DeleteTablet for tablet b2c4c7fbb70448c0bdcf52c45418ae1d with delete_type TABLET_DATA_TOMBSTONED (TS 6bd26e15a57f49c298b66adbf108b46b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39580
I20251026 08:53:23.732146 5409 tablet_replica.cc:333] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: stopping tablet replica
I20251026 08:53:23.732347 5409 raft_consensus.cc:2243] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:23.732587 5409 raft_consensus.cc:2272] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:23.733378 5409 ts_tablet_manager.cc:1916] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:23.734555 5204 consensus_queue.cc:237] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } }
I20251026 08:53:23.736501 5318 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 FOLLOWER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:53:23.736508 5054 raft_consensus.cc:1275] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Refusing update from remote peer 99f502d0d17e487f9f856f9719b324c4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:53:23.736971 5406 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:23.737233 5382 consensus_queue.cc:1048] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:23.739240 5409 ts_tablet_manager.cc:1929] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20251026 08:53:23.739365 5409 log.cc:1199] T b2c4c7fbb70448c0bdcf52c45418ae1d P 6bd26e15a57f49c298b66adbf108b46b: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-0-root/wals/b2c4c7fbb70448c0bdcf52c45418ae1d
I20251026 08:53:23.740092 4939 catalog_manager.cc:4985] TS 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761): tablet b2c4c7fbb70448c0bdcf52c45418ae1d (table test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]) successfully deleted
I20251026 08:53:23.739929 5054 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.739930 5318 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.744050 4953 catalog_manager.cc:5649] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b reported cstate change: config changed from index 3 to 4, VOTER 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "99f502d0d17e487f9f856f9719b324c4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.801216 5120 tablet_service.cc:1552] Processing DeleteTablet for tablet f572d7dd7bf1426d88ab48eaf24fc6d6 with delete_type TABLET_DATA_TOMBSTONED (TS 4095c3e315d74b57b085b807cc11f5c5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60390
I20251026 08:53:23.801867 5410 tablet_replica.cc:333] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: stopping tablet replica
I20251026 08:53:23.802151 5410 raft_consensus.cc:2243] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:23.802423 5410 raft_consensus.cc:2272] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:23.803437 5410 ts_tablet_manager.cc:1916] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:53:23.808341 5410 ts_tablet_manager.cc:1929] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20251026 08:53:23.808447 5410 log.cc:1199] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 4095c3e315d74b57b085b807cc11f5c5: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1761468669168814-413-0/minicluster-data/ts-1-root/wals/f572d7dd7bf1426d88ab48eaf24fc6d6
I20251026 08:53:23.809001 5364 raft_consensus.cc:2955] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "4095c3e315d74b57b085b807cc11f5c5" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 43541 } } peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.810259 4938 catalog_manager.cc:4985] TS 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66:43541): tablet f572d7dd7bf1426d88ab48eaf24fc6d6 (table test-workload [id=9e0bc1b1e78848dbb2be7bec9e103a0f]) successfully deleted
I20251026 08:53:23.811900 4938 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet b2c4c7fbb70448c0bdcf52c45418ae1d with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:23.839057 5406 raft_consensus.cc:2955] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4095c3e315d74b57b085b807cc11f5c5 (127.0.103.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "99f502d0d17e487f9f856f9719b324c4" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 35361 } } peers { permanent_uuid: "6bd26e15a57f49c298b66adbf108b46b" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 38761 } } peers { permanent_uuid: "dfa324dc29d844a58ef46c063cc6cd4e" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 34527 } attrs { promote: false } } }
I20251026 08:53:23.842232 4940 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet f572d7dd7bf1426d88ab48eaf24fc6d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:24.237869 5006 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:24.238638 5006 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:24.238762 5006 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:26.239179 5006 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:26.239848 5006 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:26.239951 5006 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:28.240262 5006 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:28.240888 5006 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:28.240985 5006 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:30.241288 5006 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:30.241892 5006 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:30.242000 5006 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:30.724856 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:53:30.757661 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:30.757983 413 tablet_replica.cc:333] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b: stopping tablet replica
I20251026 08:53:30.758234 413 raft_consensus.cc:2243] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:30.758504 413 raft_consensus.cc:2272] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 6bd26e15a57f49c298b66adbf108b46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.759248 413 tablet_replica.cc:333] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b: stopping tablet replica
I20251026 08:53:30.759505 413 raft_consensus.cc:2243] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:30.759819 413 raft_consensus.cc:2272] T ba836a02819540db96233ecb92595945 P 6bd26e15a57f49c298b66adbf108b46b [term 2 FOLLOWER]: Raft consensus is shut down!
W20251026 08:53:30.765175 5169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111)
W20251026 08:53:30.766211 5169 consensus_peers.cc:597] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 -> Peer 6bd26e15a57f49c298b66adbf108b46b (127.0.103.65:38761): Couldn't send request to peer 6bd26e15a57f49c298b66adbf108b46b. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:38761: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:30.774372 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:53:30.777253 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:53:30.785239 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:30.785447 413 tablet_replica.cc:333] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5: stopping tablet replica
I20251026 08:53:30.785597 413 raft_consensus.cc:2243] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:30.785843 413 raft_consensus.cc:2272] T b2c4c7fbb70448c0bdcf52c45418ae1d P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.786331 413 tablet_replica.cc:333] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5: stopping tablet replica
I20251026 08:53:30.786459 413 raft_consensus.cc:2243] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:30.786590 413 raft_consensus.cc:2272] T ba836a02819540db96233ecb92595945 P 4095c3e315d74b57b085b807cc11f5c5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.800186 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:53:30.818056 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:53:30.825736 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:30.825959 413 tablet_replica.cc:333] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4: stopping tablet replica
I20251026 08:53:30.826181 413 raft_consensus.cc:2243] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:30.826467 413 raft_consensus.cc:2272] T f572d7dd7bf1426d88ab48eaf24fc6d6 P 99f502d0d17e487f9f856f9719b324c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.826961 413 tablet_replica.cc:333] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4: stopping tablet replica
I20251026 08:53:30.827095 413 raft_consensus.cc:2243] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:30.827231 413 raft_consensus.cc:2272] T ba836a02819540db96233ecb92595945 P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.827678 413 tablet_replica.cc:333] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4: stopping tablet replica
I20251026 08:53:30.827778 413 raft_consensus.cc:2243] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:30.828007 413 raft_consensus.cc:2272] T b2c4c7fbb70448c0bdcf52c45418ae1d P 99f502d0d17e487f9f856f9719b324c4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.841720 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:53:30.844537 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:53:30.852943 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:30.853168 413 tablet_replica.cc:333] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e: stopping tablet replica
I20251026 08:53:30.853305 413 raft_consensus.cc:2243] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:30.853482 413 raft_consensus.cc:2272] T b2c4c7fbb70448c0bdcf52c45418ae1d P dfa324dc29d844a58ef46c063cc6cd4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.853883 413 tablet_replica.cc:333] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e: stopping tablet replica
I20251026 08:53:30.853994 413 raft_consensus.cc:2243] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:30.854226 413 raft_consensus.cc:2272] T f572d7dd7bf1426d88ab48eaf24fc6d6 P dfa324dc29d844a58ef46c063cc6cd4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.867161 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:53:30.869743 413 master.cc:561] Master@127.0.103.126:35743 shutting down...
I20251026 08:53:30.877874 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:30.878033 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:30.878144 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9f7113ff12024fadbbde37744a622d87: stopping tablet replica
I20251026 08:53:30.891434 413 master.cc:583] Master@127.0.103.126:35743 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (10714 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20251026 08:53:30.903261 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:42725
I20251026 08:53:30.903661 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:30.905397 5415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:30.905557 5416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:30.906162 5418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:30.906730 413 server_base.cc:1047] running on GCE node
I20251026 08:53:30.906916 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:30.906958 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:30.906980 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468810906976 us; error 0 us; skew 500 ppm
I20251026 08:53:30.907116 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:30.908207 413 webserver.cc:492] Webserver started at http://127.0.103.126:34597/ using document root <none> and password file <none>
I20251026 08:53:30.908427 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:30.908480 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:30.908596 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:30.909094 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "2c32216fb1f94d65a26a2103b0d0f419"
format_stamp: "Formatted at 2025-10-26 08:53:30 on dist-test-slave-rd83"
I20251026 08:53:30.911188 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 08:53:30.912598 5423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:30.912887 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:53:30.912984 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "2c32216fb1f94d65a26a2103b0d0f419"
format_stamp: "Formatted at 2025-10-26 08:53:30 on dist-test-slave-rd83"
I20251026 08:53:30.913081 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:30.918239 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:30.918635 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:30.928467 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:42725
I20251026 08:53:30.928491 5474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:42725 every 8 connection(s)
I20251026 08:53:30.929544 5475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:30.933348 5475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419: Bootstrap starting.
I20251026 08:53:30.934571 5475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:30.935811 5475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419: No bootstrap required, opened a new log
I20251026 08:53:30.936383 5475 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c32216fb1f94d65a26a2103b0d0f419" member_type: VOTER }
I20251026 08:53:30.936497 5475 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:30.936568 5475 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c32216fb1f94d65a26a2103b0d0f419, State: Initialized, Role: FOLLOWER
I20251026 08:53:30.936750 5475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c32216fb1f94d65a26a2103b0d0f419" member_type: VOTER }
I20251026 08:53:30.936853 5475 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:53:30.936925 5475 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:53:30.937006 5475 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:30.938683 5475 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c32216fb1f94d65a26a2103b0d0f419" member_type: VOTER }
I20251026 08:53:30.938853 5475 leader_election.cc:304] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c32216fb1f94d65a26a2103b0d0f419; no voters:
I20251026 08:53:30.939188 5475 leader_election.cc:290] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:53:30.939277 5478 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:30.939639 5478 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 1 LEADER]: Becoming Leader. State: Replica: 2c32216fb1f94d65a26a2103b0d0f419, State: Running, Role: LEADER
I20251026 08:53:30.939860 5475 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:53:30.939827 5478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c32216fb1f94d65a26a2103b0d0f419" member_type: VOTER }
I20251026 08:53:30.940973 5480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c32216fb1f94d65a26a2103b0d0f419. Latest consensus state: current_term: 1 leader_uuid: "2c32216fb1f94d65a26a2103b0d0f419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c32216fb1f94d65a26a2103b0d0f419" member_type: VOTER } }
I20251026 08:53:30.941188 5480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:30.941921 5486 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:53:30.941875 5479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c32216fb1f94d65a26a2103b0d0f419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c32216fb1f94d65a26a2103b0d0f419" member_type: VOTER } }
I20251026 08:53:30.942078 5479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:30.943145 5486 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:53:30.944455 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:53:30.945804 5486 catalog_manager.cc:1357] Generated new cluster ID: 7821d3824e0a47588cd089353e337f94
I20251026 08:53:30.945883 5486 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:53:30.977308 5486 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:53:30.977933 5486 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:53:30.985752 5486 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419: Generated new TSK 0
I20251026 08:53:30.985930 5486 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:53:31.009356 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:31.011742 5496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:31.011926 5497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:31.013115 5499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:31.013152 413 server_base.cc:1047] running on GCE node
I20251026 08:53:31.013442 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:31.013495 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:31.013538 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468811013534 us; error 0 us; skew 500 ppm
I20251026 08:53:31.013692 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:31.014799 413 webserver.cc:492] Webserver started at http://127.0.103.65:42413/ using document root <none> and password file <none>
I20251026 08:53:31.014993 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:31.015070 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:31.015161 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:31.015700 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "6aa6911853b54546bca1208669fd15b8"
format_stamp: "Formatted at 2025-10-26 08:53:31 on dist-test-slave-rd83"
I20251026 08:53:31.018054 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:31.019433 5504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.019721 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:31.019814 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "6aa6911853b54546bca1208669fd15b8"
format_stamp: "Formatted at 2025-10-26 08:53:31 on dist-test-slave-rd83"
I20251026 08:53:31.019948 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:31.028121 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:31.028561 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:31.029023 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:31.029825 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:31.029881 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.029947 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:31.029987 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.041687 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:33465
I20251026 08:53:31.041704 5566 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:33465 every 8 connection(s)
I20251026 08:53:31.043334 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:31.045590 5571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:31.045625 5572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:31.046762 5574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:31.049261 5567 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42725
I20251026 08:53:31.049371 5567 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:31.049628 5567 heartbeater.cc:507] Master 127.0.103.126:42725 requested a full tablet report, sending...
I20251026 08:53:31.050359 5440 ts_manager.cc:194] Registered new tserver with Master: 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465)
I20251026 08:53:31.051242 5440 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58238
I20251026 08:53:31.052438 413 server_base.cc:1047] running on GCE node
I20251026 08:53:31.052608 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:31.052655 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:31.052692 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468811052689 us; error 0 us; skew 500 ppm
I20251026 08:53:31.052801 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:31.053707 413 webserver.cc:492] Webserver started at http://127.0.103.66:46559/ using document root <none> and password file <none>
I20251026 08:53:31.053880 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:31.053936 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:31.054037 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:31.054567 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "4241dae1691149ae8f9dbfbd14fcdd81"
format_stamp: "Formatted at 2025-10-26 08:53:31 on dist-test-slave-rd83"
I20251026 08:53:31.056404 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:31.057595 5579 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.057899 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:31.057967 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "4241dae1691149ae8f9dbfbd14fcdd81"
format_stamp: "Formatted at 2025-10-26 08:53:31 on dist-test-slave-rd83"
I20251026 08:53:31.058049 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:31.065356 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:31.065683 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:31.066047 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:31.066653 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:31.066695 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.066730 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:31.066754 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.078179 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:37839
I20251026 08:53:31.078251 5641 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:37839 every 8 connection(s)
I20251026 08:53:31.079614 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:31.082319 5646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:31.082388 5645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:31.083662 5648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:31.083879 413 server_base.cc:1047] running on GCE node
I20251026 08:53:31.084331 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:31.084417 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:31.084487 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468811084462 us; error 0 us; skew 500 ppm
I20251026 08:53:31.084653 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:31.085810 5642 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42725
I20251026 08:53:31.085839 413 webserver.cc:492] Webserver started at http://127.0.103.67:38705/ using document root <none> and password file <none>
I20251026 08:53:31.085968 5642 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:31.086058 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:31.086201 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:31.086393 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:31.086536 5642 heartbeater.cc:507] Master 127.0.103.126:42725 requested a full tablet report, sending...
I20251026 08:53:31.087023 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "615ef9255b62431e9eed6fe367d383b7"
format_stamp: "Formatted at 2025-10-26 08:53:31 on dist-test-slave-rd83"
I20251026 08:53:31.087441 5440 ts_manager.cc:194] Registered new tserver with Master: 4241dae1691149ae8f9dbfbd14fcdd81 (127.0.103.66:37839)
I20251026 08:53:31.088434 5440 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58242
I20251026 08:53:31.089737 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:53:31.091176 5653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.091481 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:31.091580 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "615ef9255b62431e9eed6fe367d383b7"
format_stamp: "Formatted at 2025-10-26 08:53:31 on dist-test-slave-rd83"
I20251026 08:53:31.091683 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:31.124747 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:31.125157 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:31.125586 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:31.126310 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:31.126385 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.126477 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:31.126533 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:31.138419 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:38187
I20251026 08:53:31.138458 5715 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:38187 every 8 connection(s)
I20251026 08:53:31.144199 5716 heartbeater.cc:344] Connected to a master server at 127.0.103.126:42725
I20251026 08:53:31.144306 5716 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:31.144536 5716 heartbeater.cc:507] Master 127.0.103.126:42725 requested a full tablet report, sending...
I20251026 08:53:31.145179 5440 ts_manager.cc:194] Registered new tserver with Master: 615ef9255b62431e9eed6fe367d383b7 (127.0.103.67:38187)
I20251026 08:53:31.145753 5440 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58244
I20251026 08:53:31.145942 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006748551s
I20251026 08:53:32.052312 5567 heartbeater.cc:499] Master 127.0.103.126:42725 was elected leader, sending a full tablet report...
I20251026 08:53:32.089551 5642 heartbeater.cc:499] Master 127.0.103.126:42725 was elected leader, sending a full tablet report...
I20251026 08:53:32.146761 5716 heartbeater.cc:499] Master 127.0.103.126:42725 was elected leader, sending a full tablet report...
I20251026 08:53:32.172767 413 test_util.cc:276] Using random seed: -355131943
I20251026 08:53:32.187512 5440 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58252:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:53:32.188033 5440 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:53:32.223978 5681 tablet_service.cc:1505] Processing CreateTablet for tablet 8d448c3482fb4625bc57dd71974be23a (DEFAULT_TABLE table=test-workload [id=b132384abba74ad2a02aacacb38f2553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:53:32.224471 5681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d448c3482fb4625bc57dd71974be23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:32.225003 5607 tablet_service.cc:1505] Processing CreateTablet for tablet 8d448c3482fb4625bc57dd71974be23a (DEFAULT_TABLE table=test-workload [id=b132384abba74ad2a02aacacb38f2553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:53:32.225306 5607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d448c3482fb4625bc57dd71974be23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:32.226205 5532 tablet_service.cc:1505] Processing CreateTablet for tablet 8d448c3482fb4625bc57dd71974be23a (DEFAULT_TABLE table=test-workload [id=b132384abba74ad2a02aacacb38f2553]), partition=RANGE (key) PARTITION UNBOUNDED
I20251026 08:53:32.226585 5532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d448c3482fb4625bc57dd71974be23a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:32.229434 5736 tablet_bootstrap.cc:492] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81: Bootstrap starting.
I20251026 08:53:32.230047 5737 tablet_bootstrap.cc:492] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7: Bootstrap starting.
I20251026 08:53:32.231271 5736 tablet_bootstrap.cc:654] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:32.231964 5737 tablet_bootstrap.cc:654] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:32.232352 5738 tablet_bootstrap.cc:492] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8: Bootstrap starting.
I20251026 08:53:32.233763 5736 tablet_bootstrap.cc:492] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81: No bootstrap required, opened a new log
I20251026 08:53:32.233942 5736 ts_tablet_manager.cc:1403] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251026 08:53:32.234028 5738 tablet_bootstrap.cc:654] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:32.234714 5736 raft_consensus.cc:359] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.234889 5736 raft_consensus.cc:385] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:32.234969 5736 raft_consensus.cc:740] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4241dae1691149ae8f9dbfbd14fcdd81, State: Initialized, Role: FOLLOWER
I20251026 08:53:32.235051 5737 tablet_bootstrap.cc:492] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7: No bootstrap required, opened a new log
I20251026 08:53:32.235157 5737 ts_tablet_manager.cc:1403] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251026 08:53:32.235184 5736 consensus_queue.cc:260] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.235857 5736 ts_tablet_manager.cc:1434] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:32.235816 5737 raft_consensus.cc:359] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.235965 5737 raft_consensus.cc:385] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:32.236044 5737 raft_consensus.cc:740] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615ef9255b62431e9eed6fe367d383b7, State: Initialized, Role: FOLLOWER
I20251026 08:53:32.236248 5737 consensus_queue.cc:260] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.236898 5737 ts_tablet_manager.cc:1434] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:53:32.238495 5738 tablet_bootstrap.cc:492] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8: No bootstrap required, opened a new log
I20251026 08:53:32.238638 5738 ts_tablet_manager.cc:1403] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20251026 08:53:32.239179 5738 raft_consensus.cc:359] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.239313 5738 raft_consensus.cc:385] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:32.239392 5738 raft_consensus.cc:740] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aa6911853b54546bca1208669fd15b8, State: Initialized, Role: FOLLOWER
I20251026 08:53:32.239600 5738 consensus_queue.cc:260] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.240211 5738 ts_tablet_manager.cc:1434] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:53:32.246963 5743 raft_consensus.cc:493] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:32.247078 5743 raft_consensus.cc:515] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.247632 5743 leader_election.cc:290] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465), 4241dae1691149ae8f9dbfbd14fcdd81 (127.0.103.66:37839)
I20251026 08:53:32.254055 5617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d448c3482fb4625bc57dd71974be23a" candidate_uuid: "615ef9255b62431e9eed6fe367d383b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" is_pre_election: true
I20251026 08:53:32.254096 5542 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d448c3482fb4625bc57dd71974be23a" candidate_uuid: "615ef9255b62431e9eed6fe367d383b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa6911853b54546bca1208669fd15b8" is_pre_election: true
I20251026 08:53:32.254330 5542 raft_consensus.cc:2468] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615ef9255b62431e9eed6fe367d383b7 in term 0.
I20251026 08:53:32.254330 5617 raft_consensus.cc:2468] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615ef9255b62431e9eed6fe367d383b7 in term 0.
I20251026 08:53:32.254720 5654 leader_election.cc:304] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615ef9255b62431e9eed6fe367d383b7, 6aa6911853b54546bca1208669fd15b8; no voters:
I20251026 08:53:32.254984 5743 raft_consensus.cc:2804] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:32.255112 5743 raft_consensus.cc:493] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:32.255209 5743 raft_consensus.cc:3060] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:32.256798 5743 raft_consensus.cc:515] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.257247 5743 leader_election.cc:290] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [CANDIDATE]: Term 1 election: Requested vote from peers 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465), 4241dae1691149ae8f9dbfbd14fcdd81 (127.0.103.66:37839)
I20251026 08:53:32.257541 5542 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d448c3482fb4625bc57dd71974be23a" candidate_uuid: "615ef9255b62431e9eed6fe367d383b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6aa6911853b54546bca1208669fd15b8"
I20251026 08:53:32.257581 5617 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d448c3482fb4625bc57dd71974be23a" candidate_uuid: "615ef9255b62431e9eed6fe367d383b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4241dae1691149ae8f9dbfbd14fcdd81"
I20251026 08:53:32.257665 5542 raft_consensus.cc:3060] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:32.257776 5617 raft_consensus.cc:3060] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:32.259353 5542 raft_consensus.cc:2468] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615ef9255b62431e9eed6fe367d383b7 in term 1.
I20251026 08:53:32.259505 5617 raft_consensus.cc:2468] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615ef9255b62431e9eed6fe367d383b7 in term 1.
I20251026 08:53:32.259694 5654 leader_election.cc:304] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615ef9255b62431e9eed6fe367d383b7, 6aa6911853b54546bca1208669fd15b8; no voters:
I20251026 08:53:32.259958 5743 raft_consensus.cc:2804] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:32.260310 5743 raft_consensus.cc:697] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 1 LEADER]: Becoming Leader. State: Replica: 615ef9255b62431e9eed6fe367d383b7, State: Running, Role: LEADER
I20251026 08:53:32.260547 5743 consensus_queue.cc:237] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } }
I20251026 08:53:32.262607 5440 catalog_manager.cc:5649] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615ef9255b62431e9eed6fe367d383b7 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "615ef9255b62431e9eed6fe367d383b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6aa6911853b54546bca1208669fd15b8" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 33465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615ef9255b62431e9eed6fe367d383b7" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 38187 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:32.268309 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:53:32.275471 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:32.275771 413 tablet_replica.cc:333] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8: stopping tablet replica
I20251026 08:53:32.275918 413 raft_consensus.cc:2243] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:32.276055 413 raft_consensus.cc:2272] T 8d448c3482fb4625bc57dd71974be23a P 6aa6911853b54546bca1208669fd15b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:32.288895 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:53:32.697770 5743 consensus_queue.cc:1048] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4241dae1691149ae8f9dbfbd14fcdd81" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 37839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:53:32.703251 5654 consensus_peers.cc:597] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 -> Peer 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465): Couldn't send request to peer 6aa6911853b54546bca1208669fd15b8. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:32.944348 5493 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:32.944741 5493 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:32.944855 5493 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:34.945263 5493 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:34.945762 5493 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:34.945888 5493 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 08:53:35.258491 5654 consensus_peers.cc:597] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 -> Peer 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465): Couldn't send request to peer 6aa6911853b54546bca1208669fd15b8. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20251026 08:53:35.881873 5654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20251026 08:53:36.946303 5493 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:36.946789 5493 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:36.946887 5493 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 08:53:37.914770 5654 consensus_peers.cc:597] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 -> Peer 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465): Couldn't send request to peer 6aa6911853b54546bca1208669fd15b8. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251026 08:53:38.947310 5493 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:38.947793 5493 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:38.947908 5493 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 08:53:40.520776 5654 consensus_peers.cc:597] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 -> Peer 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465): Couldn't send request to peer 6aa6911853b54546bca1208669fd15b8. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20251026 08:53:40.948267 5493 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:40.948676 5493 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:40.948784 5493 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20251026 08:53:40.969103 5654 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251026 08:53:42.892096 5654 consensus_peers.cc:597] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 -> Peer 6aa6911853b54546bca1208669fd15b8 (127.0.103.65:33465): Couldn't send request to peer 6aa6911853b54546bca1208669fd15b8. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:33465: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20251026 08:53:42.949132 5493 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20251026 08:53:42.949597 5493 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20251026 08:53:42.949705 5493 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20251026 08:53:43.344612 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:53:43.362964 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:43.363308 413 tablet_replica.cc:333] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81: stopping tablet replica
I20251026 08:53:43.363505 413 raft_consensus.cc:2243] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:43.363682 413 raft_consensus.cc:2272] T 8d448c3482fb4625bc57dd71974be23a P 4241dae1691149ae8f9dbfbd14fcdd81 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:43.377933 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:53:43.380265 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:53:43.387452 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:43.387650 413 tablet_replica.cc:333] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7: stopping tablet replica
I20251026 08:53:43.387804 413 raft_consensus.cc:2243] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:43.388054 413 raft_consensus.cc:2272] T 8d448c3482fb4625bc57dd71974be23a P 615ef9255b62431e9eed6fe367d383b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:43.417197 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:53:43.419768 413 master.cc:561] Master@127.0.103.126:42725 shutting down...
I20251026 08:53:43.427646 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:43.427831 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:43.427914 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 2c32216fb1f94d65a26a2103b0d0f419: stopping tablet replica
I20251026 08:53:43.440659 413 master.cc:583] Master@127.0.103.126:42725 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (12546 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20251026 08:53:43.450040 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:46709
I20251026 08:53:43.450443 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:43.452659 5782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.452646 5779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.452627 5780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:43.453047 413 server_base.cc:1047] running on GCE node
I20251026 08:53:43.453490 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:43.453542 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:43.453562 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468823453559 us; error 0 us; skew 500 ppm
I20251026 08:53:43.453696 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:43.454717 413 webserver.cc:492] Webserver started at http://127.0.103.126:36233/ using document root <none> and password file <none>
I20251026 08:53:43.454895 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:43.454949 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:43.455046 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:43.455603 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "63d7c287abde41ad9f334a483461b0d4"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.457762 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:43.459034 5787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.459368 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:43.459437 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "63d7c287abde41ad9f334a483461b0d4"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.459524 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:43.467926 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:43.468326 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:43.477694 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:46709
I20251026 08:53:43.477711 5838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:46709 every 8 connection(s)
I20251026 08:53:43.478752 5839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:43.482197 5839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4: Bootstrap starting.
I20251026 08:53:43.483374 5839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:43.484731 5839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4: No bootstrap required, opened a new log
I20251026 08:53:43.485208 5839 raft_consensus.cc:359] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7c287abde41ad9f334a483461b0d4" member_type: VOTER }
I20251026 08:53:43.485302 5839 raft_consensus.cc:385] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:43.485358 5839 raft_consensus.cc:740] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63d7c287abde41ad9f334a483461b0d4, State: Initialized, Role: FOLLOWER
I20251026 08:53:43.485519 5839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7c287abde41ad9f334a483461b0d4" member_type: VOTER }
I20251026 08:53:43.485606 5839 raft_consensus.cc:399] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:53:43.485657 5839 raft_consensus.cc:493] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:53:43.485723 5839 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:43.487126 5839 raft_consensus.cc:515] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7c287abde41ad9f334a483461b0d4" member_type: VOTER }
I20251026 08:53:43.487280 5839 leader_election.cc:304] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63d7c287abde41ad9f334a483461b0d4; no voters:
I20251026 08:53:43.487591 5839 leader_election.cc:290] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:53:43.487666 5842 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:43.488032 5842 raft_consensus.cc:697] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 1 LEADER]: Becoming Leader. State: Replica: 63d7c287abde41ad9f334a483461b0d4, State: Running, Role: LEADER
I20251026 08:53:43.488262 5839 sys_catalog.cc:565] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:53:43.488224 5842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7c287abde41ad9f334a483461b0d4" member_type: VOTER }
I20251026 08:53:43.488814 5844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63d7c287abde41ad9f334a483461b0d4. Latest consensus state: current_term: 1 leader_uuid: "63d7c287abde41ad9f334a483461b0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7c287abde41ad9f334a483461b0d4" member_type: VOTER } }
I20251026 08:53:43.488793 5843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63d7c287abde41ad9f334a483461b0d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63d7c287abde41ad9f334a483461b0d4" member_type: VOTER } }
I20251026 08:53:43.488958 5844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:43.489034 5843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:43.489903 5847 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:53:43.491609 5847 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:53:43.492137 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:53:43.494119 5847 catalog_manager.cc:1357] Generated new cluster ID: be2470caa1a94d05beed3028f2038f3a
I20251026 08:53:43.494184 5847 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:53:43.516839 5847 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:53:43.517503 5847 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:53:43.525933 5847 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4: Generated new TSK 0
I20251026 08:53:43.526129 5847 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:53:43.557061 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:43.559679 5860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.559922 5863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.560014 5861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:43.560674 413 server_base.cc:1047] running on GCE node
I20251026 08:53:43.560938 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:43.561002 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:43.561034 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468823561030 us; error 0 us; skew 500 ppm
I20251026 08:53:43.561173 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:43.563733 413 webserver.cc:492] Webserver started at http://127.0.103.65:34229/ using document root <none> and password file <none>
I20251026 08:53:43.563942 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:43.564010 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:43.564136 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:43.564859 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "119a60f3147845f8af40cfaca471496e"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.567624 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:53:43.569067 5868 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.569458 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:43.569537 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "119a60f3147845f8af40cfaca471496e"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.569638 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:43.585815 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:43.586206 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:43.586591 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:43.587255 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:43.587298 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.587344 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:43.587370 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.598605 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:35077
I20251026 08:53:43.598619 5930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:35077 every 8 connection(s)
I20251026 08:53:43.599985 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:43.602381 5936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.602371 5935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.604086 5938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:43.604561 413 server_base.cc:1047] running on GCE node
I20251026 08:53:43.604800 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:43.604846 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:43.604867 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468823604864 us; error 0 us; skew 500 ppm
I20251026 08:53:43.605005 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:43.606142 413 webserver.cc:492] Webserver started at http://127.0.103.66:35267/ using document root <none> and password file <none>
I20251026 08:53:43.606359 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:43.606405 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:43.606504 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:43.607045 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "0a8d5b3457db49e7b67a75c1ff8969d2"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.608973 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 08:53:43.610191 5943 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.610447 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:53:43.610524 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "0a8d5b3457db49e7b67a75c1ff8969d2"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.610611 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:43.613301 5931 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46709
I20251026 08:53:43.613433 5931 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:43.613718 5931 heartbeater.cc:507] Master 127.0.103.126:46709 requested a full tablet report, sending...
I20251026 08:53:43.614460 5804 ts_manager.cc:194] Registered new tserver with Master: 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077)
I20251026 08:53:43.615320 5804 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42818
I20251026 08:53:43.626387 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:43.626802 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:43.627208 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:43.627832 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:43.627877 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.627962 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:43.627995 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.639021 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:34903
I20251026 08:53:43.639042 6005 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:34903 every 8 connection(s)
I20251026 08:53:43.640427 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:43.643268 6009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.643530 6012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:43.644089 6010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:43.644713 413 server_base.cc:1047] running on GCE node
I20251026 08:53:43.644942 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:43.645025 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:43.645083 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468823645062 us; error 0 us; skew 500 ppm
I20251026 08:53:43.645246 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:43.646173 6006 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46709
I20251026 08:53:43.646319 6006 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:43.646347 413 webserver.cc:492] Webserver started at http://127.0.103.67:43405/ using document root <none> and password file <none>
I20251026 08:53:43.646538 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:43.646612 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:43.646588 6006 heartbeater.cc:507] Master 127.0.103.126:46709 requested a full tablet report, sending...
I20251026 08:53:43.646742 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:43.647379 5804 ts_manager.cc:194] Registered new tserver with Master: 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:43.647383 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "3b52f17979d14782b12009d80d4d13e3"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.648362 5804 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42834
I20251026 08:53:43.649950 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:43.651335 6017 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.651659 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:43.651765 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "3b52f17979d14782b12009d80d4d13e3"
format_stamp: "Formatted at 2025-10-26 08:53:43 on dist-test-slave-rd83"
I20251026 08:53:43.651854 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:43.658650 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:43.659050 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:43.659452 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:43.660051 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:43.660120 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.660223 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:43.660281 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:43.671044 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44763
I20251026 08:53:43.671070 6079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44763 every 8 connection(s)
I20251026 08:53:43.676095 6080 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46709
I20251026 08:53:43.676198 6080 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:43.676453 6080 heartbeater.cc:507] Master 127.0.103.126:46709 requested a full tablet report, sending...
I20251026 08:53:43.677102 5804 ts_manager.cc:194] Registered new tserver with Master: 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:43.677299 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.005596763s
I20251026 08:53:43.677982 5804 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42844
I20251026 08:53:44.616447 5931 heartbeater.cc:499] Master 127.0.103.126:46709 was elected leader, sending a full tablet report...
I20251026 08:53:44.649556 6006 heartbeater.cc:499] Master 127.0.103.126:46709 was elected leader, sending a full tablet report...
I20251026 08:53:44.679136 6080 heartbeater.cc:499] Master 127.0.103.126:46709 was elected leader, sending a full tablet report...
I20251026 08:53:44.704666 413 test_util.cc:276] Using random seed: -342600044
I20251026 08:53:44.719453 5804 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42858:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:53:44.720072 5804 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:53:44.744787 6045 tablet_service.cc:1505] Processing CreateTablet for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:53:44.744905 5971 tablet_service.cc:1505] Processing CreateTablet for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:53:44.745229 5971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.745229 6045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.746493 6042 tablet_service.cc:1505] Processing CreateTablet for tablet 10207a3878644976b9f7f7e18e525162 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:53:44.746856 6042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10207a3878644976b9f7f7e18e525162. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.749432 5896 tablet_service.cc:1505] Processing CreateTablet for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:53:44.749738 5970 tablet_service.cc:1505] Processing CreateTablet for tablet de59faf6cc8840ff99b12c1adcaa5002 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:53:44.750120 5970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de59faf6cc8840ff99b12c1adcaa5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.750169 5895 tablet_service.cc:1505] Processing CreateTablet for tablet de59faf6cc8840ff99b12c1adcaa5002 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:53:44.750753 5894 tablet_service.cc:1505] Processing CreateTablet for tablet 9d41d6d3f7c6491382e10f5c592235d2 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:53:44.751233 5893 tablet_service.cc:1505] Processing CreateTablet for tablet 10207a3878644976b9f7f7e18e525162 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:53:44.749825 5896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.753161 6100 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3: Bootstrap starting.
I20251026 08:53:44.754428 6043 tablet_service.cc:1505] Processing CreateTablet for tablet 9d41d6d3f7c6491382e10f5c592235d2 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:53:44.754774 6043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d41d6d3f7c6491382e10f5c592235d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.754833 6100 tablet_bootstrap.cc:654] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.757309 5893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10207a3878644976b9f7f7e18e525162. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.758471 6044 tablet_service.cc:1505] Processing CreateTablet for tablet de59faf6cc8840ff99b12c1adcaa5002 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:53:44.758975 6044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de59faf6cc8840ff99b12c1adcaa5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.759685 5894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d41d6d3f7c6491382e10f5c592235d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.762217 5895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de59faf6cc8840ff99b12c1adcaa5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.762521 5968 tablet_service.cc:1505] Processing CreateTablet for tablet 10207a3878644976b9f7f7e18e525162 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:53:44.763007 5968 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10207a3878644976b9f7f7e18e525162. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.754429 5969 tablet_service.cc:1505] Processing CreateTablet for tablet 9d41d6d3f7c6491382e10f5c592235d2 (DEFAULT_TABLE table=test-workload [id=e2f6607e77f440278ecced22ae91cfb6]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:53:44.767190 5969 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d41d6d3f7c6491382e10f5c592235d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:44.786696 6103 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: Bootstrap starting.
I20251026 08:53:44.788993 6103 tablet_bootstrap.cc:654] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.791167 6101 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: Bootstrap starting.
I20251026 08:53:44.792425 6100 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3: No bootstrap required, opened a new log
I20251026 08:53:44.792445 6103 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: No bootstrap required, opened a new log
I20251026 08:53:44.792659 6100 ts_tablet_manager.cc:1403] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3: Time spent bootstrapping tablet: real 0.040s user 0.019s sys 0.019s
I20251026 08:53:44.792666 6103 ts_tablet_manager.cc:1403] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20251026 08:53:44.793362 6103 raft_consensus.cc:359] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.793542 6103 raft_consensus.cc:385] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.793502 6100 raft_consensus.cc:359] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.793635 6101 tablet_bootstrap.cc:654] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.793648 6103 raft_consensus.cc:740] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.793730 6100 raft_consensus.cc:385] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.793813 6100 raft_consensus.cc:740] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b52f17979d14782b12009d80d4d13e3, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.794027 6103 consensus_queue.cc:260] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.794128 6100 consensus_queue.cc:260] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.794898 6103 ts_tablet_manager.cc:1434] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:44.795512 6103 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: Bootstrap starting.
I20251026 08:53:44.797158 6103 tablet_bootstrap.cc:654] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.795192 6100 ts_tablet_manager.cc:1434] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:44.798341 6100 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: Bootstrap starting.
I20251026 08:53:44.799575 6101 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: No bootstrap required, opened a new log
I20251026 08:53:44.799803 6101 ts_tablet_manager.cc:1403] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20251026 08:53:44.800531 6101 raft_consensus.cc:359] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.800673 6101 raft_consensus.cc:385] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.800761 6101 raft_consensus.cc:740] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.800840 6100 tablet_bootstrap.cc:654] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.800977 6101 consensus_queue.cc:260] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.802160 6101 ts_tablet_manager.cc:1434] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:44.802418 6103 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: No bootstrap required, opened a new log
I20251026 08:53:44.802464 6101 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: Bootstrap starting.
I20251026 08:53:44.802661 6103 ts_tablet_manager.cc:1403] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.002s
I20251026 08:53:44.803648 6103 raft_consensus.cc:359] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.803869 6103 raft_consensus.cc:385] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.803979 6103 raft_consensus.cc:740] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.804124 6101 tablet_bootstrap.cc:654] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.804260 6103 consensus_queue.cc:260] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.805498 6103 ts_tablet_manager.cc:1434] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:53:44.805835 6103 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e: Bootstrap starting.
I20251026 08:53:44.807772 6103 tablet_bootstrap.cc:654] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.809692 6100 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: No bootstrap required, opened a new log
I20251026 08:53:44.809902 6100 ts_tablet_manager.cc:1403] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20251026 08:53:44.810642 6100 raft_consensus.cc:359] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.810765 6100 raft_consensus.cc:385] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.810846 6100 raft_consensus.cc:740] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b52f17979d14782b12009d80d4d13e3, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.811033 6100 consensus_queue.cc:260] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.811498 6103 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e: No bootstrap required, opened a new log
I20251026 08:53:44.811612 6100 ts_tablet_manager.cc:1434] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:53:44.811756 6103 ts_tablet_manager.cc:1403] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20251026 08:53:44.811874 6100 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3: Bootstrap starting.
I20251026 08:53:44.812706 6103 raft_consensus.cc:359] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.812968 6103 raft_consensus.cc:385] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.813064 6103 raft_consensus.cc:740] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.813338 6103 consensus_queue.cc:260] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.813525 6100 tablet_bootstrap.cc:654] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.813927 6103 ts_tablet_manager.cc:1434] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:53:44.814774 6103 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: Bootstrap starting.
I20251026 08:53:44.815902 6101 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: No bootstrap required, opened a new log
I20251026 08:53:44.816083 6101 ts_tablet_manager.cc:1403] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.010s
I20251026 08:53:44.816612 6103 tablet_bootstrap.cc:654] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.816726 6101 raft_consensus.cc:359] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.816960 6101 raft_consensus.cc:385] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.817080 6101 raft_consensus.cc:740] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.817310 6101 consensus_queue.cc:260] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.817708 6101 ts_tablet_manager.cc:1434] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 08:53:44.817952 6101 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2: Bootstrap starting.
I20251026 08:53:44.819438 6101 tablet_bootstrap.cc:654] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.821391 6100 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3: No bootstrap required, opened a new log
I20251026 08:53:44.821620 6100 ts_tablet_manager.cc:1403] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.006s
I20251026 08:53:44.822582 6100 raft_consensus.cc:359] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.822813 6100 raft_consensus.cc:385] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.822916 6100 raft_consensus.cc:740] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b52f17979d14782b12009d80d4d13e3, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.823180 6101 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2: No bootstrap required, opened a new log
I20251026 08:53:44.823187 6100 consensus_queue.cc:260] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.823390 6101 ts_tablet_manager.cc:1403] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20251026 08:53:44.823496 6105 raft_consensus.cc:493] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:44.823660 6105 raft_consensus.cc:515] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.823814 6100 ts_tablet_manager.cc:1434] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:44.824051 6100 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3: Bootstrap starting.
I20251026 08:53:44.824194 6101 raft_consensus.cc:359] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.824330 6101 raft_consensus.cc:385] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.824414 6101 raft_consensus.cc:740] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.824524 6105 leader_election.cc:290] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763), 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:44.824648 6101 consensus_queue.cc:260] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.825451 6100 tablet_bootstrap.cc:654] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.825771 6101 ts_tablet_manager.cc:1434] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:53:44.826110 6101 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: Bootstrap starting.
I20251026 08:53:44.828128 6101 tablet_bootstrap.cc:654] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:44.838905 6103 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: No bootstrap required, opened a new log
I20251026 08:53:44.839152 6103 ts_tablet_manager.cc:1403] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.008s
I20251026 08:53:44.840081 6103 raft_consensus.cc:359] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.840428 6103 raft_consensus.cc:385] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.840590 6103 raft_consensus.cc:740] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.840916 6103 consensus_queue.cc:260] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.843422 6103 ts_tablet_manager.cc:1434] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251026 08:53:44.847203 6101 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: No bootstrap required, opened a new log
I20251026 08:53:44.847364 6101 ts_tablet_manager.cc:1403] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.017s
I20251026 08:53:44.847640 6100 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3: No bootstrap required, opened a new log
I20251026 08:53:44.847833 6100 ts_tablet_manager.cc:1403] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.007s
I20251026 08:53:44.848165 6101 raft_consensus.cc:359] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.848399 6101 raft_consensus.cc:385] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.848526 6101 raft_consensus.cc:740] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.848587 6100 raft_consensus.cc:359] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.848810 6100 raft_consensus.cc:385] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:44.848925 6100 raft_consensus.cc:740] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b52f17979d14782b12009d80d4d13e3, State: Initialized, Role: FOLLOWER
I20251026 08:53:44.848810 6101 consensus_queue.cc:260] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.849239 6100 consensus_queue.cc:260] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.849563 6101 ts_tablet_manager.cc:1434] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:53:44.849743 6100 ts_tablet_manager.cc:1434] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:53:44.852234 6055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0ac4a6ec1a4a289c133fe08b2398c7" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3" is_pre_election: true
I20251026 08:53:44.852347 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0ac4a6ec1a4a289c133fe08b2398c7" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" is_pre_election: true
I20251026 08:53:44.852481 6055 raft_consensus.cc:2468] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 0.
I20251026 08:53:44.852559 5981 raft_consensus.cc:2468] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 0.
I20251026 08:53:44.853108 5870 leader_election.cc:304] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a60f3147845f8af40cfaca471496e, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:44.853385 6105 raft_consensus.cc:2804] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:44.853505 6105 raft_consensus.cc:493] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:44.853619 6105 raft_consensus.cc:3060] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.855722 6105 raft_consensus.cc:515] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.856122 6105 leader_election.cc:290] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 election: Requested vote from peers 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763), 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:44.856410 6055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0ac4a6ec1a4a289c133fe08b2398c7" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3"
I20251026 08:53:44.856478 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0ac4a6ec1a4a289c133fe08b2398c7" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2"
I20251026 08:53:44.856560 6055 raft_consensus.cc:3060] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.856590 6105 raft_consensus.cc:493] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:44.856694 5981 raft_consensus.cc:3060] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.856731 6105 raft_consensus.cc:515] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.857223 6105 leader_election.cc:290] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:44.857542 5980 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10207a3878644976b9f7f7e18e525162" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" is_pre_election: true
I20251026 08:53:44.857725 5980 raft_consensus.cc:2468] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 0.
I20251026 08:53:44.857802 6105 raft_consensus.cc:493] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:44.857942 6105 raft_consensus.cc:515] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.858494 6105 leader_election.cc:290] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:44.858721 6055 raft_consensus.cc:2468] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 1.
I20251026 08:53:44.859090 6055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d41d6d3f7c6491382e10f5c592235d2" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3" is_pre_election: true
I20251026 08:53:44.859179 5981 raft_consensus.cc:2468] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 1.
I20251026 08:53:44.859237 6055 raft_consensus.cc:2468] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 0.
I20251026 08:53:44.859261 5870 leader_election.cc:304] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a60f3147845f8af40cfaca471496e, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:44.859536 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10207a3878644976b9f7f7e18e525162" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3" is_pre_election: true
I20251026 08:53:44.859733 6105 raft_consensus.cc:2804] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:44.859750 6054 raft_consensus.cc:2468] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 0.
I20251026 08:53:44.860234 5870 leader_election.cc:304] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a60f3147845f8af40cfaca471496e, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:44.860447 5872 leader_election.cc:304] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 119a60f3147845f8af40cfaca471496e; no voters:
I20251026 08:53:44.860633 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d41d6d3f7c6491382e10f5c592235d2" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" is_pre_election: true
I20251026 08:53:44.860862 5981 raft_consensus.cc:2468] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 0.
I20251026 08:53:44.860826 6105 raft_consensus.cc:697] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: Becoming Leader. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Running, Role: LEADER
I20251026 08:53:44.861515 6105 consensus_queue.cc:237] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:44.860822 6120 raft_consensus.cc:2804] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:44.862484 6109 raft_consensus.cc:493] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:44.862649 6109 raft_consensus.cc:515] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.863667 6109 leader_election.cc:290] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:44.867033 6120 raft_consensus.cc:493] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:44.867269 6120 raft_consensus.cc:3060] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.870397 6120 raft_consensus.cc:515] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.871161 6120 leader_election.cc:290] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 election: Requested vote from peers 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:44.871397 6120 raft_consensus.cc:2804] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:44.871537 6120 raft_consensus.cc:493] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:44.871670 6120 raft_consensus.cc:3060] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.874845 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d41d6d3f7c6491382e10f5c592235d2" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2"
I20251026 08:53:44.875032 5981 raft_consensus.cc:3060] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.874994 6120 raft_consensus.cc:515] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.875705 6120 leader_election.cc:290] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 election: Requested vote from peers 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:44.876688 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d41d6d3f7c6491382e10f5c592235d2" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3"
I20251026 08:53:44.876848 6054 raft_consensus.cc:3060] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.876812 6055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10207a3878644976b9f7f7e18e525162" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3"
I20251026 08:53:44.877022 6055 raft_consensus.cc:3060] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.877960 5981 raft_consensus.cc:2468] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 1.
I20251026 08:53:44.878676 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10207a3878644976b9f7f7e18e525162" candidate_uuid: "119a60f3147845f8af40cfaca471496e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2"
I20251026 08:53:44.878875 5981 raft_consensus.cc:3060] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.879302 6055 raft_consensus.cc:2468] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 1.
I20251026 08:53:44.879352 6054 raft_consensus.cc:2468] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 1.
I20251026 08:53:44.879756 5870 leader_election.cc:304] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a60f3147845f8af40cfaca471496e, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:44.879995 6121 raft_consensus.cc:2804] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:44.880038 5870 leader_election.cc:304] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a60f3147845f8af40cfaca471496e, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:44.880398 6122 raft_consensus.cc:2804] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:44.880518 6122 raft_consensus.cc:697] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: Becoming Leader. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Running, Role: LEADER
I20251026 08:53:44.880759 6122 consensus_queue.cc:237] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.880220 6121 raft_consensus.cc:697] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: Becoming Leader. State: Replica: 119a60f3147845f8af40cfaca471496e, State: Running, Role: LEADER
I20251026 08:53:44.881762 5981 raft_consensus.cc:2468] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a60f3147845f8af40cfaca471496e in term 1.
I20251026 08:53:44.881920 6121 consensus_queue.cc:237] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.883147 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de59faf6cc8840ff99b12c1adcaa5002" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3" is_pre_election: true
I20251026 08:53:44.883371 6054 raft_consensus.cc:2468] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 0.
I20251026 08:53:44.883814 5945 leader_election.cc:304] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:44.884037 6109 raft_consensus.cc:2804] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:44.884130 6109 raft_consensus.cc:493] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:44.884208 6109 raft_consensus.cc:3060] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.884558 5906 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de59faf6cc8840ff99b12c1adcaa5002" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119a60f3147845f8af40cfaca471496e" is_pre_election: true
I20251026 08:53:44.884768 5906 raft_consensus.cc:2468] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 0.
I20251026 08:53:44.885628 6109 raft_consensus.cc:515] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.885982 6109 leader_election.cc:290] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 1 election: Requested vote from peers 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:44.886648 5906 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de59faf6cc8840ff99b12c1adcaa5002" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119a60f3147845f8af40cfaca471496e"
I20251026 08:53:44.886833 5906 raft_consensus.cc:3060] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.887804 5804 catalog_manager.cc:5649] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e reported cstate change: term changed from 0 to 1, leader changed from <none> to 119a60f3147845f8af40cfaca471496e (127.0.103.65). New cstate: current_term: 1 leader_uuid: "119a60f3147845f8af40cfaca471496e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:44.888547 5906 raft_consensus.cc:2468] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 1.
I20251026 08:53:44.888863 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "de59faf6cc8840ff99b12c1adcaa5002" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3"
I20251026 08:53:44.888900 5947 leader_election.cc:304] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 119a60f3147845f8af40cfaca471496e; no voters:
I20251026 08:53:44.889055 6054 raft_consensus.cc:3060] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:44.889146 6109 raft_consensus.cc:2804] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:44.889585 6109 raft_consensus.cc:697] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 LEADER]: Becoming Leader. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Running, Role: LEADER
I20251026 08:53:44.889838 6109 consensus_queue.cc:237] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:44.891198 6054 raft_consensus.cc:2468] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 1.
I20251026 08:53:44.892385 5804 catalog_manager.cc:5649] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e reported cstate change: term changed from 0 to 1, leader changed from <none> to 119a60f3147845f8af40cfaca471496e (127.0.103.65). New cstate: current_term: 1 leader_uuid: "119a60f3147845f8af40cfaca471496e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:44.892827 5804 catalog_manager.cc:5649] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e reported cstate change: term changed from 0 to 1, leader changed from <none> to 119a60f3147845f8af40cfaca471496e (127.0.103.65). New cstate: current_term: 1 leader_uuid: "119a60f3147845f8af40cfaca471496e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:44.893359 5803 catalog_manager.cc:5649] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:44.931592 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:53:44.938031 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:44.938447 413 tablet_replica.cc:333] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e: stopping tablet replica
I20251026 08:53:44.938611 413 raft_consensus.cc:2243] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:44.938786 413 raft_consensus.cc:2272] T de59faf6cc8840ff99b12c1adcaa5002 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:44.939361 413 tablet_replica.cc:333] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: stopping tablet replica
I20251026 08:53:44.939498 413 raft_consensus.cc:2243] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:44.939695 413 pending_rounds.cc:70] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: Trying to abort 1 pending ops.
I20251026 08:53:44.939754 413 pending_rounds.cc:77] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976306718924800 op_type: NO_OP noop_request { }
I20251026 08:53:44.939865 413 raft_consensus.cc:2889] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:44.951897 413 raft_consensus.cc:2272] T 10207a3878644976b9f7f7e18e525162 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:44.952682 413 tablet_replica.cc:333] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: stopping tablet replica
I20251026 08:53:44.952811 413 raft_consensus.cc:2243] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:44.953006 413 pending_rounds.cc:70] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: Trying to abort 1 pending ops.
I20251026 08:53:44.953051 413 pending_rounds.cc:77] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976306636615680 op_type: NO_OP noop_request { }
I20251026 08:53:44.953147 413 raft_consensus.cc:2889] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:44.953228 413 raft_consensus.cc:2272] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:44.953840 413 tablet_replica.cc:333] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: stopping tablet replica
I20251026 08:53:44.953953 413 raft_consensus.cc:2243] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:44.954150 413 pending_rounds.cc:70] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: Trying to abort 1 pending ops.
I20251026 08:53:44.954190 413 pending_rounds.cc:77] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976306713714688 op_type: NO_OP noop_request { }
I20251026 08:53:44.954257 413 raft_consensus.cc:2889] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:44.954342 413 raft_consensus.cc:2272] T 9d41d6d3f7c6491382e10f5c592235d2 P 119a60f3147845f8af40cfaca471496e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:44.968660 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:53:45.357693 6109 consensus_queue.cc:1048] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:53:45.363101 5947 consensus_peers.cc:597] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:45.972608 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:45.975030 6136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:45.975036 6137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:45.975620 6139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:45.975857 413 server_base.cc:1047] running on GCE node
I20251026 08:53:45.976061 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:45.976107 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:45.976146 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468825976143 us; error 0 us; skew 500 ppm
I20251026 08:53:45.976277 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:45.977466 413 webserver.cc:492] Webserver started at http://127.0.103.68:44979/ using document root <none> and password file <none>
I20251026 08:53:45.977677 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:45.977744 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:45.977857 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:45.978554 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "6a29d2dd75684a46a2940dc167a92d08"
format_stamp: "Formatted at 2025-10-26 08:53:45 on dist-test-slave-rd83"
I20251026 08:53:45.980815 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20251026 08:53:45.982223 6144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:45.982568 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:53:45.982682 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "6a29d2dd75684a46a2940dc167a92d08"
format_stamp: "Formatted at 2025-10-26 08:53:45 on dist-test-slave-rd83"
I20251026 08:53:45.982771 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:45.987746 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:45.988129 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:45.988572 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:45.989266 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:45.989316 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:45.989388 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:45.989430 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:46.000937 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:41067
I20251026 08:53:46.000959 6206 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:41067 every 8 connection(s)
I20251026 08:53:46.007033 6207 heartbeater.cc:344] Connected to a master server at 127.0.103.126:46709
I20251026 08:53:46.007126 6207 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:46.007373 6207 heartbeater.cc:507] Master 127.0.103.126:46709 requested a full tablet report, sending...
I20251026 08:53:46.008018 5804 ts_manager.cc:194] Registered new tserver with Master: 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68:41067)
I20251026 08:53:46.008680 5804 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42866
I20251026 08:53:46.241593 6133 consensus_queue.cc:579] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.351s)
W20251026 08:53:46.242875 5947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20251026 08:53:46.245446 5981 consensus_queue.cc:237] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:46.247520 6054 raft_consensus.cc:1275] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:53:46.248051 6212 consensus_queue.cc:1048] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 08:53:46.249320 5947 consensus_peers.cc:597] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:46.249802 6133 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:46.250398 6054 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:46.253247 5791 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet de59faf6cc8840ff99b12c1adcaa5002 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20251026 08:53:46.254087 5945 consensus_peers.cc:597] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68:41067): Couldn't send request to peer 6a29d2dd75684a46a2940dc167a92d08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de59faf6cc8840ff99b12c1adcaa5002. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:46.254575 5804 catalog_manager.cc:5649] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 08:53:46.493810 5852 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 119a60f3147845f8af40cfaca471496e not available for placement
I20251026 08:53:46.719406 6211 raft_consensus.cc:493] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:46.719583 6211 raft_consensus.cc:515] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:46.720131 6211 leader_election.cc:290] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:46.720606 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d41d6d3f7c6491382e10f5c592235d2" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3" is_pre_election: true
I20251026 08:53:46.720770 6054 raft_consensus.cc:2468] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 1.
I20251026 08:53:46.721150 5945 leader_election.cc:304] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters:
W20251026 08:53:46.721417 5947 leader_election.cc:336] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:46.721441 6211 raft_consensus.cc:2804] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:53:46.721524 6211 raft_consensus.cc:493] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:46.721593 6211 raft_consensus.cc:3060] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:46.723555 6211 raft_consensus.cc:515] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:46.723973 6211 leader_election.cc:290] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 election: Requested vote from peers 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077), 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:46.724402 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9d41d6d3f7c6491382e10f5c592235d2" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3"
I20251026 08:53:46.724578 6054 raft_consensus.cc:3060] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Advancing to term 2
W20251026 08:53:46.725047 5947 leader_election.cc:336] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:46.726483 6054 raft_consensus.cc:2468] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 2.
I20251026 08:53:46.726792 5945 leader_election.cc:304] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters: 119a60f3147845f8af40cfaca471496e
I20251026 08:53:46.726989 6211 raft_consensus.cc:2804] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:53:46.727097 6211 raft_consensus.cc:697] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Running, Role: LEADER
I20251026 08:53:46.727344 6211 consensus_queue.cc:237] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:46.729462 5803 catalog_manager.cc:5649] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: term changed from 1 to 2, leader changed from 119a60f3147845f8af40cfaca471496e (127.0.103.65) to 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:46.780836 6221 raft_consensus.cc:493] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:46.781009 6221 raft_consensus.cc:515] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:46.781908 6221 leader_election.cc:290] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077), 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:46.794724 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10207a3878644976b9f7f7e18e525162" candidate_uuid: "3b52f17979d14782b12009d80d4d13e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" is_pre_election: true
I20251026 08:53:46.794893 5981 raft_consensus.cc:2468] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b52f17979d14782b12009d80d4d13e3 in term 1.
W20251026 08:53:46.794935 6021 leader_election.cc:336] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:46.795215 6021 leader_election.cc:304] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters: 119a60f3147845f8af40cfaca471496e
I20251026 08:53:46.795457 6221 raft_consensus.cc:2804] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:53:46.795535 6221 raft_consensus.cc:493] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:46.795604 6221 raft_consensus.cc:3060] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:53:46.798003 6221 raft_consensus.cc:515] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:46.798547 6221 leader_election.cc:290] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [CANDIDATE]: Term 2 election: Requested vote from peers 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077), 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:46.799350 5981 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "10207a3878644976b9f7f7e18e525162" candidate_uuid: "3b52f17979d14782b12009d80d4d13e3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2"
I20251026 08:53:46.799515 5981 raft_consensus.cc:3060] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Advancing to term 2
W20251026 08:53:46.799790 6021 leader_election.cc:336] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:46.802153 5981 raft_consensus.cc:2468] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b52f17979d14782b12009d80d4d13e3 in term 2.
I20251026 08:53:46.802416 6021 leader_election.cc:304] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters: 119a60f3147845f8af40cfaca471496e
I20251026 08:53:46.802701 6221 raft_consensus.cc:2804] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:53:46.803122 6221 raft_consensus.cc:697] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 LEADER]: Becoming Leader. State: Replica: 3b52f17979d14782b12009d80d4d13e3, State: Running, Role: LEADER
I20251026 08:53:46.803481 6221 consensus_queue.cc:237] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } }
I20251026 08:53:46.804383 6226 ts_tablet_manager.cc:933] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: Initiating tablet copy from peer 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:46.804941 6226 tablet_copy_client.cc:323] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34903
I20251026 08:53:46.806334 5803 catalog_manager.cc:5649] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 reported cstate change: term changed from 1 to 2, leader changed from 119a60f3147845f8af40cfaca471496e (127.0.103.65) to 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "3b52f17979d14782b12009d80d4d13e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:46.811690 5991 tablet_copy_service.cc:140] P 0a8d5b3457db49e7b67a75c1ff8969d2: Received BeginTabletCopySession request for tablet de59faf6cc8840ff99b12c1adcaa5002 from peer 6a29d2dd75684a46a2940dc167a92d08 ({username='slave'} at 127.0.0.1:49228)
I20251026 08:53:46.811841 5991 tablet_copy_service.cc:161] P 0a8d5b3457db49e7b67a75c1ff8969d2: Beginning new tablet copy session on tablet de59faf6cc8840ff99b12c1adcaa5002 from peer 6a29d2dd75684a46a2940dc167a92d08 at {username='slave'} at 127.0.0.1:49228: session id = 6a29d2dd75684a46a2940dc167a92d08-de59faf6cc8840ff99b12c1adcaa5002
I20251026 08:53:46.815156 5991 tablet_copy_source_session.cc:215] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:46.816226 6226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de59faf6cc8840ff99b12c1adcaa5002. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:46.821086 6226 tablet_copy_client.cc:806] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:46.821278 6226 tablet_copy_client.cc:670] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:46.824407 6226 tablet_copy_client.cc:538] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:46.826946 6226 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap starting.
I20251026 08:53:46.831275 6226 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:46.831588 6226 tablet_bootstrap.cc:492] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap complete.
I20251026 08:53:46.831758 6226 ts_tablet_manager.cc:1403] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:53:46.832274 6226 raft_consensus.cc:359] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:46.832433 6226 raft_consensus.cc:740] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6a29d2dd75684a46a2940dc167a92d08, State: Initialized, Role: LEARNER
I20251026 08:53:46.832579 6226 consensus_queue.cc:260] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:46.833179 6207 heartbeater.cc:499] Master 127.0.103.126:46709 was elected leader, sending a full tablet report...
I20251026 08:53:46.833177 6226 ts_tablet_manager.cc:1434] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:53:46.833889 5991 tablet_copy_service.cc:342] P 0a8d5b3457db49e7b67a75c1ff8969d2: Request end of tablet copy session 6a29d2dd75684a46a2940dc167a92d08-de59faf6cc8840ff99b12c1adcaa5002 received from {username='slave'} at 127.0.0.1:49228
I20251026 08:53:46.833998 5991 tablet_copy_service.cc:434] P 0a8d5b3457db49e7b67a75c1ff8969d2: ending tablet copy session 6a29d2dd75684a46a2940dc167a92d08-de59faf6cc8840ff99b12c1adcaa5002 on tablet de59faf6cc8840ff99b12c1adcaa5002 with peer 6a29d2dd75684a46a2940dc167a92d08
I20251026 08:53:46.943653 6211 raft_consensus.cc:493] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:46.943840 6211 raft_consensus.cc:515] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:46.944398 6211 leader_election.cc:290] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763), 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077)
I20251026 08:53:46.944850 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0ac4a6ec1a4a289c133fe08b2398c7" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3" is_pre_election: true
I20251026 08:53:46.945019 6054 raft_consensus.cc:2468] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 1.
I20251026 08:53:46.945335 5945 leader_election.cc:304] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters:
I20251026 08:53:46.945595 6211 raft_consensus.cc:2804] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:53:46.945686 6211 raft_consensus.cc:493] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:46.945761 6211 raft_consensus.cc:3060] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Advancing to term 2
W20251026 08:53:46.945744 5947 leader_election.cc:336] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:46.947849 6211 raft_consensus.cc:515] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:46.948243 6211 leader_election.cc:290] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 election: Requested vote from peers 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763), 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077)
I20251026 08:53:46.948583 6054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1a0ac4a6ec1a4a289c133fe08b2398c7" candidate_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b52f17979d14782b12009d80d4d13e3"
I20251026 08:53:46.948720 6054 raft_consensus.cc:3060] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Advancing to term 2
W20251026 08:53:46.949265 5947 leader_election.cc:336] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:46.950920 6054 raft_consensus.cc:2468] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a8d5b3457db49e7b67a75c1ff8969d2 in term 2.
I20251026 08:53:46.951197 5945 leader_election.cc:304] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0a8d5b3457db49e7b67a75c1ff8969d2, 3b52f17979d14782b12009d80d4d13e3; no voters: 119a60f3147845f8af40cfaca471496e
I20251026 08:53:46.951407 6211 raft_consensus.cc:2804] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:53:46.951511 6211 raft_consensus.cc:697] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Becoming Leader. State: Replica: 0a8d5b3457db49e7b67a75c1ff8969d2, State: Running, Role: LEADER
I20251026 08:53:46.951704 6211 consensus_queue.cc:237] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } }
I20251026 08:53:46.953824 5803 catalog_manager.cc:5649] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: term changed from 1 to 2, leader changed from 119a60f3147845f8af40cfaca471496e (127.0.103.65) to 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66). New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } }
W20251026 08:53:47.117389 5947 consensus_peers.cc:597] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:47.220844 6054 raft_consensus.cc:1275] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 08:53:47.221400 6211 consensus_queue.cc:1048] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:53:47.254417 6021 consensus_peers.cc:597] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:47.297807 6211 consensus_queue.cc:579] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.052s)
I20251026 08:53:47.313855 6182 raft_consensus.cc:1217] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 08:53:47.400497 6054 raft_consensus.cc:1275] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 08:53:47.401007 6236 consensus_queue.cc:1048] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:53:47.405220 5947 consensus_peers.cc:597] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:47.429160 5981 raft_consensus.cc:1275] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Refusing update from remote peer 3b52f17979d14782b12009d80d4d13e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20251026 08:53:47.429631 6221 consensus_queue.cc:1048] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:53:47.494446 5852 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 119a60f3147845f8af40cfaca471496e not available for placement
I20251026 08:53:47.930027 6240 raft_consensus.cc:1064] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: attempting to promote NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 to VOTER
I20251026 08:53:47.930713 6240 consensus_queue.cc:237] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:47.932507 6182 raft_consensus.cc:1275] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 LEARNER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:53:47.932505 6054 raft_consensus.cc:1275] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:53:47.933157 6236 consensus_queue.cc:1048] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:47.933398 6242 consensus_queue.cc:1048] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 08:53:47.933928 5947 consensus_peers.cc:597] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:47.935765 6242 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:47.936172 6054 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:47.938966 6182 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:47.939967 5804 catalog_manager.cc:5649] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:47.944715 5981 consensus_queue.cc:237] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:47.946098 6182 raft_consensus.cc:1275] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:53:47.946151 6054 raft_consensus.cc:1275] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:53:47.946527 6245 consensus_queue.cc:1048] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:47.946640 6242 consensus_queue.cc:1048] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:47.948220 6236 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:47.948710 6054 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:47.948926 6182 raft_consensus.cc:2955] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:47.951344 5791 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet de59faf6cc8840ff99b12c1adcaa5002 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:47.952841 5802 catalog_manager.cc:5649] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251026 08:53:47.956633 5791 catalog_manager.cc:4712] TS 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet de59faf6cc8840ff99b12c1adcaa5002: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:48.025875 6242 consensus_queue.cc:579] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.074s)
I20251026 08:53:48.028853 5981 consensus_queue.cc:237] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.030524 6054 raft_consensus.cc:1275] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 08:53:48.030666 5945 consensus_peers.cc:597] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68:41067): Couldn't send request to peer 6a29d2dd75684a46a2940dc167a92d08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a0ac4a6ec1a4a289c133fe08b2398c7. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:48.031018 6240 consensus_queue.cc:1048] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20251026 08:53:48.031147 5947 consensus_peers.cc:597] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:48.032605 6245 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:48.033210 6054 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:48.035395 5791 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 08:53:48.036656 5804 catalog_manager.cc:5649] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:48.146379 6240 consensus_queue.cc:579] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.419s)
I20251026 08:53:48.149554 5981 consensus_queue.cc:237] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.151273 6054 raft_consensus.cc:1275] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20251026 08:53:48.151338 5945 consensus_peers.cc:597] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68:41067): Couldn't send request to peer 6a29d2dd75684a46a2940dc167a92d08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d41d6d3f7c6491382e10f5c592235d2. This is attempt 1: this message will repeat every 5th retry.
W20251026 08:53:48.151825 5947 consensus_peers.cc:597] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:48.151860 6242 consensus_queue.cc:1048] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:48.153729 6236 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:48.154358 6054 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:48.156705 5791 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9d41d6d3f7c6491382e10f5c592235d2 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 08:53:48.158123 5803 catalog_manager.cc:5649] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:48.243664 6221 consensus_queue.cc:579] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.440s)
I20251026 08:53:48.247964 6054 consensus_queue.cc:237] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.250034 5981 raft_consensus.cc:1275] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Refusing update from remote peer 3b52f17979d14782b12009d80d4d13e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:53:48.250865 6252 consensus_queue.cc:1048] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:53:48.253077 6253 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
I20251026 08:53:48.256352 5981 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } } }
W20251026 08:53:48.257064 6021 consensus_peers.cc:597] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:48.257007 5804 catalog_manager.cc:5649] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 reported cstate change: config changed from index -1 to 2, NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "3b52f17979d14782b12009d80d4d13e3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:48.261689 5789 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 10207a3878644976b9f7f7e18e525162 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20251026 08:53:48.262261 6019 consensus_peers.cc:597] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 -> Peer 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68:41067): Couldn't send request to peer 6a29d2dd75684a46a2940dc167a92d08. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10207a3878644976b9f7f7e18e525162. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:48.437712 6258 ts_tablet_manager.cc:933] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: Initiating tablet copy from peer 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:48.438269 6258 tablet_copy_client.cc:323] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34903
I20251026 08:53:48.438814 5991 tablet_copy_service.cc:140] P 0a8d5b3457db49e7b67a75c1ff8969d2: Received BeginTabletCopySession request for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 from peer 6a29d2dd75684a46a2940dc167a92d08 ({username='slave'} at 127.0.0.1:49228)
I20251026 08:53:48.438928 5991 tablet_copy_service.cc:161] P 0a8d5b3457db49e7b67a75c1ff8969d2: Beginning new tablet copy session on tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 from peer 6a29d2dd75684a46a2940dc167a92d08 at {username='slave'} at 127.0.0.1:49228: session id = 6a29d2dd75684a46a2940dc167a92d08-1a0ac4a6ec1a4a289c133fe08b2398c7
I20251026 08:53:48.440614 5991 tablet_copy_source_session.cc:215] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:48.441629 6258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:48.445573 6258 tablet_copy_client.cc:806] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:48.445732 6258 tablet_copy_client.cc:670] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:48.447085 6258 tablet_copy_client.cc:538] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:48.448916 6258 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap starting.
I20251026 08:53:48.453390 6258 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:48.453685 6258 tablet_bootstrap.cc:492] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap complete.
I20251026 08:53:48.453887 6258 ts_tablet_manager.cc:1403] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20251026 08:53:48.454412 6258 raft_consensus.cc:359] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.454576 6258 raft_consensus.cc:740] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6a29d2dd75684a46a2940dc167a92d08, State: Initialized, Role: LEARNER
I20251026 08:53:48.454733 6258 consensus_queue.cc:260] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.455209 6258 ts_tablet_manager.cc:1434] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20251026 08:53:48.455734 5991 tablet_copy_service.cc:342] P 0a8d5b3457db49e7b67a75c1ff8969d2: Request end of tablet copy session 6a29d2dd75684a46a2940dc167a92d08-1a0ac4a6ec1a4a289c133fe08b2398c7 received from {username='slave'} at 127.0.0.1:49228
I20251026 08:53:48.455857 5991 tablet_copy_service.cc:434] P 0a8d5b3457db49e7b67a75c1ff8969d2: ending tablet copy session 6a29d2dd75684a46a2940dc167a92d08-1a0ac4a6ec1a4a289c133fe08b2398c7 on tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 with peer 6a29d2dd75684a46a2940dc167a92d08
W20251026 08:53:48.495055 5852 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 119a60f3147845f8af40cfaca471496e not available for placement
I20251026 08:53:48.729700 6258 ts_tablet_manager.cc:933] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: Initiating tablet copy from peer 0a8d5b3457db49e7b67a75c1ff8969d2 (127.0.103.66:34903)
I20251026 08:53:48.730276 6258 tablet_copy_client.cc:323] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:34903
I20251026 08:53:48.730788 5991 tablet_copy_service.cc:140] P 0a8d5b3457db49e7b67a75c1ff8969d2: Received BeginTabletCopySession request for tablet 9d41d6d3f7c6491382e10f5c592235d2 from peer 6a29d2dd75684a46a2940dc167a92d08 ({username='slave'} at 127.0.0.1:49228)
I20251026 08:53:48.730919 5991 tablet_copy_service.cc:161] P 0a8d5b3457db49e7b67a75c1ff8969d2: Beginning new tablet copy session on tablet 9d41d6d3f7c6491382e10f5c592235d2 from peer 6a29d2dd75684a46a2940dc167a92d08 at {username='slave'} at 127.0.0.1:49228: session id = 6a29d2dd75684a46a2940dc167a92d08-9d41d6d3f7c6491382e10f5c592235d2
I20251026 08:53:48.732649 5991 tablet_copy_source_session.cc:215] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:48.733675 6258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d41d6d3f7c6491382e10f5c592235d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:48.737594 6258 tablet_copy_client.cc:806] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:48.737778 6258 tablet_copy_client.cc:670] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:48.738808 6258 tablet_copy_client.cc:538] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:48.740680 6258 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap starting.
I20251026 08:53:48.744697 6258 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:48.744926 6258 tablet_bootstrap.cc:492] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap complete.
I20251026 08:53:48.745077 6258 ts_tablet_manager.cc:1403] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.004s
I20251026 08:53:48.745530 6258 raft_consensus.cc:359] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.745659 6258 raft_consensus.cc:740] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6a29d2dd75684a46a2940dc167a92d08, State: Initialized, Role: LEARNER
I20251026 08:53:48.745774 6258 consensus_queue.cc:260] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.746165 6258 ts_tablet_manager.cc:1434] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:53:48.746680 5991 tablet_copy_service.cc:342] P 0a8d5b3457db49e7b67a75c1ff8969d2: Request end of tablet copy session 6a29d2dd75684a46a2940dc167a92d08-9d41d6d3f7c6491382e10f5c592235d2 received from {username='slave'} at 127.0.0.1:49228
I20251026 08:53:48.746788 5991 tablet_copy_service.cc:434] P 0a8d5b3457db49e7b67a75c1ff8969d2: ending tablet copy session 6a29d2dd75684a46a2940dc167a92d08-9d41d6d3f7c6491382e10f5c592235d2 on tablet 9d41d6d3f7c6491382e10f5c592235d2 with peer 6a29d2dd75684a46a2940dc167a92d08
I20251026 08:53:48.872030 6258 ts_tablet_manager.cc:933] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: Initiating tablet copy from peer 3b52f17979d14782b12009d80d4d13e3 (127.0.103.67:44763)
I20251026 08:53:48.872494 6258 tablet_copy_client.cc:323] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44763
I20251026 08:53:48.877650 6065 tablet_copy_service.cc:140] P 3b52f17979d14782b12009d80d4d13e3: Received BeginTabletCopySession request for tablet 10207a3878644976b9f7f7e18e525162 from peer 6a29d2dd75684a46a2940dc167a92d08 ({username='slave'} at 127.0.0.1:60240)
I20251026 08:53:48.877820 6065 tablet_copy_service.cc:161] P 3b52f17979d14782b12009d80d4d13e3: Beginning new tablet copy session on tablet 10207a3878644976b9f7f7e18e525162 from peer 6a29d2dd75684a46a2940dc167a92d08 at {username='slave'} at 127.0.0.1:60240: session id = 6a29d2dd75684a46a2940dc167a92d08-10207a3878644976b9f7f7e18e525162
I20251026 08:53:48.879627 6065 tablet_copy_source_session.cc:215] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:53:48.880628 6258 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10207a3878644976b9f7f7e18e525162. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:48.884604 6258 tablet_copy_client.cc:806] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 0 data blocks...
I20251026 08:53:48.884820 6258 tablet_copy_client.cc:670] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Starting download of 1 WAL segments...
I20251026 08:53:48.886092 6258 tablet_copy_client.cc:538] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:53:48.887941 6258 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap starting.
I20251026 08:53:48.892270 6258 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:53:48.892554 6258 tablet_bootstrap.cc:492] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: Bootstrap complete.
I20251026 08:53:48.892768 6258 ts_tablet_manager.cc:1403] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20251026 08:53:48.893329 6258 raft_consensus.cc:359] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.893482 6258 raft_consensus.cc:740] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6a29d2dd75684a46a2940dc167a92d08, State: Initialized, Role: LEARNER
I20251026 08:53:48.893675 6258 consensus_queue.cc:260] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: true } }
I20251026 08:53:48.894042 6258 ts_tablet_manager.cc:1434] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:53:48.894613 6065 tablet_copy_service.cc:342] P 3b52f17979d14782b12009d80d4d13e3: Request end of tablet copy session 6a29d2dd75684a46a2940dc167a92d08-10207a3878644976b9f7f7e18e525162 received from {username='slave'} at 127.0.0.1:60240
I20251026 08:53:48.894721 6065 tablet_copy_service.cc:434] P 3b52f17979d14782b12009d80d4d13e3: ending tablet copy session 6a29d2dd75684a46a2940dc167a92d08-10207a3878644976b9f7f7e18e525162 on tablet 10207a3878644976b9f7f7e18e525162 with peer 6a29d2dd75684a46a2940dc167a92d08
I20251026 08:53:48.913980 6182 raft_consensus.cc:1217] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20251026 08:53:49.159339 5791 catalog_manager.cc:4712] TS 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet de59faf6cc8840ff99b12c1adcaa5002: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:49.162005 6271 consensus_queue.cc:579] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.012s)
I20251026 08:53:49.221721 6182 raft_consensus.cc:1217] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20251026 08:53:49.339700 6252 consensus_queue.cc:579] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.091s)
I20251026 08:53:49.466154 6182 raft_consensus.cc:1217] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20251026 08:53:49.494164 6271 consensus_queue.cc:579] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Leader has been unable to successfully communicate with peer 119a60f3147845f8af40cfaca471496e for more than 1 seconds (1.465s)
W20251026 08:53:49.495982 5852 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 119a60f3147845f8af40cfaca471496e not available for placement
I20251026 08:53:49.496727 6271 raft_consensus.cc:1064] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: attempting to promote NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 to VOTER
I20251026 08:53:49.497247 6271 consensus_queue.cc:237] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:49.498960 6054 raft_consensus.cc:1275] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:49.498992 6182 raft_consensus.cc:1275] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:49.499604 6271 consensus_queue.cc:1048] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:49.499823 6236 consensus_queue.cc:1048] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 08:53:49.500026 5947 consensus_peers.cc:597] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:49.501991 6236 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.502569 6182 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.505587 6054 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.506249 5804 catalog_manager.cc:5649] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:49.511152 5981 consensus_queue.cc:237] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:49.512581 6182 raft_consensus.cc:1275] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:49.512753 6054 raft_consensus.cc:1275] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:49.513142 6236 consensus_queue.cc:1048] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:49.513317 6270 consensus_queue.cc:1048] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:49.515591 6277 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.516080 6182 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.516194 6054 raft_consensus.cc:2955] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.519021 5802 catalog_manager.cc:5649] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:49.519302 5791 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:49.817992 6270 raft_consensus.cc:1064] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: attempting to promote NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 to VOTER
I20251026 08:53:49.818678 6270 consensus_queue.cc:237] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:49.820565 6182 raft_consensus.cc:1275] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:49.820591 6054 raft_consensus.cc:1275] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:49.821133 6270 consensus_queue.cc:1048] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:49.821429 6236 consensus_queue.cc:1048] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 08:53:49.821789 5947 consensus_peers.cc:597] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:49.824765 6270 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.825323 6182 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.829075 5804 catalog_manager.cc:5649] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 reported cstate change: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:49.829015 6054 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.833650 5981 consensus_queue.cc:237] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:49.835157 6182 raft_consensus.cc:1275] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:49.835242 6054 raft_consensus.cc:1275] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Refusing update from remote peer 0a8d5b3457db49e7b67a75c1ff8969d2: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:49.835639 6270 consensus_queue.cc:1048] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:49.836683 6270 consensus_queue.cc:1048] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:49.837544 6271 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.838151 6054 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.839003 6182 raft_consensus.cc:2955] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.841059 5791 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 9d41d6d3f7c6491382e10f5c592235d2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:49.842536 5804 catalog_manager.cc:5649] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 reported cstate change: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.875316 6275 raft_consensus.cc:1064] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: attempting to promote NON_VOTER 6a29d2dd75684a46a2940dc167a92d08 to VOTER
I20251026 08:53:49.875815 6275 consensus_queue.cc:237] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:49.877496 6182 raft_consensus.cc:1275] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 LEARNER]: Refusing update from remote peer 3b52f17979d14782b12009d80d4d13e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:49.877509 5981 raft_consensus.cc:1275] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Refusing update from remote peer 3b52f17979d14782b12009d80d4d13e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:53:49.877976 6288 consensus_queue.cc:1048] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:53:49.878203 6275 consensus_queue.cc:1048] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20251026 08:53:49.878563 6021 consensus_peers.cc:597] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 -> Peer 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): Couldn't send request to peer 119a60f3147845f8af40cfaca471496e. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:53:49.880650 6275 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.881197 6182 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.884744 5981 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.885614 5802 catalog_manager.cc:5649] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 reported cstate change: config changed from index 2 to 3, 6a29d2dd75684a46a2940dc167a92d08 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3b52f17979d14782b12009d80d4d13e3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "119a60f3147845f8af40cfaca471496e" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 35077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:53:49.889096 6054 consensus_queue.cc:237] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } }
I20251026 08:53:49.890388 5981 raft_consensus.cc:1275] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Refusing update from remote peer 3b52f17979d14782b12009d80d4d13e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:49.890389 6182 raft_consensus.cc:1275] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Refusing update from remote peer 3b52f17979d14782b12009d80d4d13e3: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20251026 08:53:49.890837 6275 consensus_queue.cc:1048] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:49.891215 6275 consensus_queue.cc:1048] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:53:49.892518 6252 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.893083 5981 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.893754 6182 raft_consensus.cc:2955] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } } }
I20251026 08:53:49.895022 5789 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 10207a3878644976b9f7f7e18e525162 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:53:49.896064 5804 catalog_manager.cc:5649] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 reported cstate change: config changed from index 3 to 4, VOTER 119a60f3147845f8af40cfaca471496e (127.0.103.65) evicted. New cstate: current_term: 2 leader_uuid: "3b52f17979d14782b12009d80d4d13e3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0a8d5b3457db49e7b67a75c1ff8969d2" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 34903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b52f17979d14782b12009d80d4d13e3" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a29d2dd75684a46a2940dc167a92d08" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 41067 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20251026 08:53:50.201689 5791 catalog_manager.cc:4712] TS 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
W20251026 08:53:51.778263 5791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20251026 08:53:51.779296 5791 catalog_manager.cc:4712] TS 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:53.028119 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:53:53.035044 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:53.035341 413 tablet_replica.cc:333] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2: stopping tablet replica
I20251026 08:53:53.035523 413 raft_consensus.cc:2243] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:53.035746 413 raft_consensus.cc:2272] T 9d41d6d3f7c6491382e10f5c592235d2 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.036242 413 tablet_replica.cc:333] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2: stopping tablet replica
I20251026 08:53:53.036365 413 raft_consensus.cc:2243] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:53.036598 413 raft_consensus.cc:2272] T de59faf6cc8840ff99b12c1adcaa5002 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.036996 413 tablet_replica.cc:333] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2: stopping tablet replica
I20251026 08:53:53.037112 413 raft_consensus.cc:2243] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.037308 413 raft_consensus.cc:2272] T 10207a3878644976b9f7f7e18e525162 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.037707 413 tablet_replica.cc:333] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2: stopping tablet replica
I20251026 08:53:53.037832 413 raft_consensus.cc:2243] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:53.038015 413 raft_consensus.cc:2272] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 0a8d5b3457db49e7b67a75c1ff8969d2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.051520 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:53:53.054848 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:53:53.062002 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:53.062261 413 tablet_replica.cc:333] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3: stopping tablet replica
I20251026 08:53:53.062408 413 raft_consensus.cc:2243] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.062624 413 raft_consensus.cc:2272] T de59faf6cc8840ff99b12c1adcaa5002 P 3b52f17979d14782b12009d80d4d13e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.063011 413 tablet_replica.cc:333] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3: stopping tablet replica
I20251026 08:53:53.063102 413 raft_consensus.cc:2243] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.063277 413 raft_consensus.cc:2272] T 9d41d6d3f7c6491382e10f5c592235d2 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.063637 413 tablet_replica.cc:333] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3: stopping tablet replica
I20251026 08:53:53.063721 413 raft_consensus.cc:2243] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.063872 413 raft_consensus.cc:2272] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.064205 413 tablet_replica.cc:333] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3: stopping tablet replica
I20251026 08:53:53.064304 413 raft_consensus.cc:2243] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:53:53.064499 413 raft_consensus.cc:2272] T 10207a3878644976b9f7f7e18e525162 P 3b52f17979d14782b12009d80d4d13e3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.077849 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:53:53.080732 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:53:53.088709 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:53.089069 413 tablet_replica.cc:333] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08: stopping tablet replica
I20251026 08:53:53.089285 413 raft_consensus.cc:2243] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.089493 413 raft_consensus.cc:2272] T 9d41d6d3f7c6491382e10f5c592235d2 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.090039 413 tablet_replica.cc:333] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08: stopping tablet replica
I20251026 08:53:53.090229 413 raft_consensus.cc:2243] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.090431 413 raft_consensus.cc:2272] T de59faf6cc8840ff99b12c1adcaa5002 P 6a29d2dd75684a46a2940dc167a92d08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.091025 413 tablet_replica.cc:333] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08: stopping tablet replica
I20251026 08:53:53.091116 413 raft_consensus.cc:2243] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.091244 413 raft_consensus.cc:2272] T 10207a3878644976b9f7f7e18e525162 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.091598 413 tablet_replica.cc:333] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08: stopping tablet replica
I20251026 08:53:53.091693 413 raft_consensus.cc:2243] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:53.091833 413 raft_consensus.cc:2272] T 1a0ac4a6ec1a4a289c133fe08b2398c7 P 6a29d2dd75684a46a2940dc167a92d08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:53.116082 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:53:53.119462 413 master.cc:561] Master@127.0.103.126:46709 shutting down...
W20251026 08:53:53.867830 5791 catalog_manager.cc:4712] TS 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a0ac4a6ec1a4a289c133fe08b2398c7: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
W20251026 08:53:56.409057 5791 catalog_manager.cc:4712] TS 119a60f3147845f8af40cfaca471496e (127.0.103.65:35077): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet de59faf6cc8840ff99b12c1adcaa5002: Network error: Client connection negotiation failed: client connection to 127.0.103.65:35077: connect: Connection refused (error 111)
I20251026 08:53:56.718741 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:56.719007 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:56.719102 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 63d7c287abde41ad9f334a483461b0d4: stopping tablet replica
I20251026 08:53:56.745113 413 master.cc:583] Master@127.0.103.126:46709 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (13308 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20251026 08:53:56.758380 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:41517
I20251026 08:53:56.758839 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:56.760810 6298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.760901 6299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.762300 6301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:56.762319 413 server_base.cc:1047] running on GCE node
I20251026 08:53:56.762589 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:56.762650 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:56.762701 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468836762686 us; error 0 us; skew 500 ppm
I20251026 08:53:56.762833 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:56.767900 413 webserver.cc:492] Webserver started at http://127.0.103.126:42815/ using document root <none> and password file <none>
I20251026 08:53:56.768095 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:56.768165 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:56.768296 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:56.768908 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "47abce3dc38645ebb6ec5b4fcc3438b7"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.771167 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:53:56.772506 6306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.772893 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:56.772984 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "47abce3dc38645ebb6ec5b4fcc3438b7"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.773077 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:56.792337 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:56.792688 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:56.801764 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:41517
I20251026 08:53:56.801784 6357 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:41517 every 8 connection(s)
I20251026 08:53:56.802868 6358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:56.806485 6358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7: Bootstrap starting.
I20251026 08:53:56.807490 6358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:56.808811 6358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7: No bootstrap required, opened a new log
I20251026 08:53:56.809355 6358 raft_consensus.cc:359] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" member_type: VOTER }
I20251026 08:53:56.809476 6358 raft_consensus.cc:385] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:56.809551 6358 raft_consensus.cc:740] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47abce3dc38645ebb6ec5b4fcc3438b7, State: Initialized, Role: FOLLOWER
I20251026 08:53:56.809747 6358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" member_type: VOTER }
I20251026 08:53:56.809866 6358 raft_consensus.cc:399] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:53:56.809937 6358 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:53:56.810020 6358 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:56.811579 6358 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" member_type: VOTER }
I20251026 08:53:56.811765 6358 leader_election.cc:304] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47abce3dc38645ebb6ec5b4fcc3438b7; no voters:
I20251026 08:53:56.812125 6358 leader_election.cc:290] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:53:56.812230 6361 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:56.812664 6361 raft_consensus.cc:697] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 1 LEADER]: Becoming Leader. State: Replica: 47abce3dc38645ebb6ec5b4fcc3438b7, State: Running, Role: LEADER
I20251026 08:53:56.812861 6358 sys_catalog.cc:565] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:53:56.812886 6361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" member_type: VOTER }
I20251026 08:53:56.813644 6363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47abce3dc38645ebb6ec5b4fcc3438b7. Latest consensus state: current_term: 1 leader_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" member_type: VOTER } }
I20251026 08:53:56.813725 6362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47abce3dc38645ebb6ec5b4fcc3438b7" member_type: VOTER } }
I20251026 08:53:56.813892 6363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:56.813998 6362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [sys.catalog]: This master's current role is: LEADER
I20251026 08:53:56.815394 6366 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:53:56.816954 6366 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:53:56.817972 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:53:56.819605 6366 catalog_manager.cc:1357] Generated new cluster ID: 689c4dea29eb4a89bab0ed0c6059f2cd
I20251026 08:53:56.819689 6366 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:53:56.842725 6366 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:53:56.843364 6366 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:53:56.854844 6366 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7: Generated new TSK 0
I20251026 08:53:56.855085 6366 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:53:56.882671 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:56.885329 6379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.885421 6380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.885954 6382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:56.886371 413 server_base.cc:1047] running on GCE node
I20251026 08:53:56.886566 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:56.886631 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:56.886682 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468836886664 us; error 0 us; skew 500 ppm
I20251026 08:53:56.886875 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:56.887841 413 webserver.cc:492] Webserver started at http://127.0.103.65:40537/ using document root <none> and password file <none>
I20251026 08:53:56.888072 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:56.888140 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:56.888260 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:56.888839 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "681cd7907543400494c8430f92a7f541"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.891006 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 08:53:56.892325 6387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.892659 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:53:56.892763 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "681cd7907543400494c8430f92a7f541"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.892876 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:56.897567 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:56.897979 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:56.898411 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:56.899325 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:56.899405 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.899534 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:56.899595 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.909638 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41857
I20251026 08:53:56.909670 6449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41857 every 8 connection(s)
I20251026 08:53:56.911118 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:56.913570 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.913800 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.914476 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:56.915176 413 server_base.cc:1047] running on GCE node
I20251026 08:53:56.915416 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:56.915482 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:56.915544 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468836915515 us; error 0 us; skew 500 ppm
I20251026 08:53:56.915709 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:56.916888 413 webserver.cc:492] Webserver started at http://127.0.103.66:37551/ using document root <none> and password file <none>
I20251026 08:53:56.917155 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:56.917248 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:56.917407 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:56.918071 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.918424 6450 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:53:56.918558 6450 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:56.918941 6450 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:53:56.919760 6323 ts_manager.cc:194] Registered new tserver with Master: 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857)
I20251026 08:53:56.920696 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20251026 08:53:56.921059 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59280
I20251026 08:53:56.922235 6462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.922536 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:53:56.922624 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.922731 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:56.932801 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:56.933174 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:56.933624 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:56.934342 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:56.934405 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.934500 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:56.934552 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.944219 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36383
I20251026 08:53:56.944248 6524 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36383 every 8 connection(s)
I20251026 08:53:56.945554 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:56.947677 6528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.947777 6529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:56.948288 6531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:56.949666 413 server_base.cc:1047] running on GCE node
I20251026 08:53:56.949919 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:56.950003 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:56.950124 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468836950088 us; error 0 us; skew 500 ppm
I20251026 08:53:56.950309 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:56.951368 413 webserver.cc:492] Webserver started at http://127.0.103.67:35563/ using document root <none> and password file <none>
I20251026 08:53:56.951608 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:56.951704 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:56.951839 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:56.951871 6525 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:53:56.952004 6525 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:56.952364 6525 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:53:56.952555 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "42b92f7783854078a49c3eee2fb0e8dc"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.953142 6323 ts_manager.cc:194] Registered new tserver with Master: aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:53:56.953806 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59292
I20251026 08:53:56.954978 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20251026 08:53:56.956164 6536 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.956418 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:56.956509 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "42b92f7783854078a49c3eee2fb0e8dc"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:53:56.956594 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:56.969702 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:56.970113 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:56.970506 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:56.971127 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:56.971194 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.971282 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:56.971343 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:56.981297 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:40531
I20251026 08:53:56.981328 6598 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:40531 every 8 connection(s)
I20251026 08:53:56.986820 6599 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:53:56.986936 6599 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:56.987219 6599 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:53:56.987881 6323 ts_manager.cc:194] Registered new tserver with Master: 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:56.988495 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59306
I20251026 08:53:56.988842 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.00675682s
I20251026 08:53:57.922269 6450 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
I20251026 08:53:57.955068 6525 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
I20251026 08:53:57.989564 6599 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
I20251026 08:53:58.015761 413 test_util.cc:276] Using random seed: -329288961
I20251026 08:53:58.026379 6323 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59312:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:53:58.026947 6323 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:53:58.046223 6415 tablet_service.cc:1505] Processing CreateTablet for tablet 43036738df554f559a3449b172705987 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:53:58.046761 6415 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43036738df554f559a3449b172705987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.050479 6414 tablet_service.cc:1505] Processing CreateTablet for tablet d32f6afe780340f5912355e73ddface9 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:53:58.050908 6414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32f6afe780340f5912355e73ddface9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.051403 6413 tablet_service.cc:1505] Processing CreateTablet for tablet 961f00cd332643dab9bccc5f28d9cf39 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:53:58.051855 6413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961f00cd332643dab9bccc5f28d9cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.051885 6412 tablet_service.cc:1505] Processing CreateTablet for tablet 0e2c96cedcd04e5289ea308fa1f0c16f (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:53:58.052330 6412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2c96cedcd04e5289ea308fa1f0c16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.058950 6619 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:53:58.059080 6564 tablet_service.cc:1505] Processing CreateTablet for tablet 43036738df554f559a3449b172705987 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:53:58.059571 6564 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43036738df554f559a3449b172705987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.060472 6619 tablet_bootstrap.cc:654] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.062323 6563 tablet_service.cc:1505] Processing CreateTablet for tablet d32f6afe780340f5912355e73ddface9 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:53:58.062516 6619 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: No bootstrap required, opened a new log
I20251026 08:53:58.062691 6563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32f6afe780340f5912355e73ddface9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.062695 6619 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:53:58.063741 6619 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.064061 6619 raft_consensus.cc:385] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.064218 6619 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.064533 6619 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.070386 6563 tablet_service.cc:1505] Processing CreateTablet for tablet 961f00cd332643dab9bccc5f28d9cf39 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:53:58.070868 6563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961f00cd332643dab9bccc5f28d9cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.071578 6619 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.009s user 0.002s sys 0.006s
I20251026 08:53:58.073411 6490 tablet_service.cc:1505] Processing CreateTablet for tablet 43036738df554f559a3449b172705987 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:53:58.073486 6489 tablet_service.cc:1505] Processing CreateTablet for tablet d32f6afe780340f5912355e73ddface9 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:53:58.073730 6619 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:53:58.073885 6490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43036738df554f559a3449b172705987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.074206 6487 tablet_service.cc:1505] Processing CreateTablet for tablet 0e2c96cedcd04e5289ea308fa1f0c16f (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:53:58.074692 6487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2c96cedcd04e5289ea308fa1f0c16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.075516 6619 tablet_bootstrap.cc:654] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.075762 6622 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:53:58.076560 6489 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32f6afe780340f5912355e73ddface9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.076772 6488 tablet_service.cc:1505] Processing CreateTablet for tablet 961f00cd332643dab9bccc5f28d9cf39 (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:53:58.077241 6488 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961f00cd332643dab9bccc5f28d9cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.077767 6622 tablet_bootstrap.cc:654] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.079560 6562 tablet_service.cc:1505] Processing CreateTablet for tablet 0e2c96cedcd04e5289ea308fa1f0c16f (DEFAULT_TABLE table=test-workload [id=7eb128f0335e4e2bb9b28a59006f425d]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:53:58.080207 6562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e2c96cedcd04e5289ea308fa1f0c16f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:53:58.093217 6619 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: No bootstrap required, opened a new log
I20251026 08:53:58.093533 6619 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20251026 08:53:58.093541 6622 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: No bootstrap required, opened a new log
I20251026 08:53:58.093792 6625 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:53:58.093961 6622 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20251026 08:53:58.094803 6619 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.094803 6622 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.095152 6619 raft_consensus.cc:385] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.095163 6622 raft_consensus.cc:385] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.095278 6619 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.095281 6622 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.095357 6625 tablet_bootstrap.cc:654] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.095520 6619 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.095522 6622 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.096114 6619 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:58.096438 6619 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:53:58.098305 6619 tablet_bootstrap.cc:654] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.101212 6622 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20251026 08:53:58.101672 6622 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:53:58.104066 6622 tablet_bootstrap.cc:654] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.101243 6625 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: No bootstrap required, opened a new log
I20251026 08:53:58.105289 6625 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.004s
I20251026 08:53:58.106333 6625 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.106516 6625 raft_consensus.cc:385] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.106599 6625 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.106921 6622 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: No bootstrap required, opened a new log
I20251026 08:53:58.106822 6625 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.107054 6622 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 08:53:58.107503 6625 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:58.107760 6625 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:53:58.108282 6619 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: No bootstrap required, opened a new log
I20251026 08:53:58.108462 6619 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20251026 08:53:58.109283 6619 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.107760 6622 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.109549 6619 raft_consensus.cc:385] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.109625 6622 raft_consensus.cc:385] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.109634 6619 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.109725 6622 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.109411 6625 tablet_bootstrap.cc:654] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.109938 6622 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.110605 6622 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:53:58.110926 6622 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:53:58.111732 6625 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: No bootstrap required, opened a new log
I20251026 08:53:58.111891 6625 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:53:58.112489 6622 tablet_bootstrap.cc:654] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.112603 6625 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.112802 6625 raft_consensus.cc:385] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.112895 6625 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.113171 6625 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.109938 6619 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.113688 6625 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:58.113893 6619 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 08:53:58.113974 6625 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:53:58.114243 6619 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:53:58.115938 6625 tablet_bootstrap.cc:654] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.115991 6619 tablet_bootstrap.cc:654] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.117414 6622 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: No bootstrap required, opened a new log
I20251026 08:53:58.117655 6622 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251026 08:53:58.118605 6625 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: No bootstrap required, opened a new log
I20251026 08:53:58.118733 6625 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251026 08:53:58.119407 6625 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.119606 6625 raft_consensus.cc:385] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.119691 6625 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.119920 6625 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.120357 6625 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251026 08:53:58.120633 6625 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:53:58.122150 6625 tablet_bootstrap.cc:654] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.122483 6622 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.123103 6619 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: No bootstrap required, opened a new log
I20251026 08:53:58.123447 6619 ts_tablet_manager.cc:1403] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20251026 08:53:58.123973 6625 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: No bootstrap required, opened a new log
I20251026 08:53:58.124111 6625 ts_tablet_manager.cc:1403] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251026 08:53:58.123317 6622 raft_consensus.cc:385] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.124450 6622 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.124739 6622 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.124807 6625 raft_consensus.cc:359] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.125015 6625 raft_consensus.cc:385] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.125085 6625 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.125216 6622 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251026 08:53:58.125270 6625 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.125687 6625 ts_tablet_manager.cc:1434] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251026 08:53:58.124230 6619 raft_consensus.cc:359] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.126178 6619 raft_consensus.cc:385] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.126289 6619 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.126510 6619 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.127107 6619 ts_tablet_manager.cc:1434] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251026 08:53:58.137838 6622 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:53:58.138733 6622 tablet_bootstrap.cc:654] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Neither blocks nor log segments found. Creating new log.
I20251026 08:53:58.139881 6622 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: No bootstrap required, opened a new log
I20251026 08:53:58.139981 6622 ts_tablet_manager.cc:1403] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:53:58.140452 6622 raft_consensus.cc:359] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.140609 6622 raft_consensus.cc:385] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:53:58.140709 6622 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:53:58.140904 6622 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.141295 6622 ts_tablet_manager.cc:1434] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20251026 08:53:58.161499 6451 tablet.cc:2378] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251026 08:53:58.162636 6621 raft_consensus.cc:493] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:58.162735 6621 raft_consensus.cc:515] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.163349 6621 leader_election.cc:290] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:58.167960 6626 raft_consensus.cc:493] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:58.168079 6626 raft_consensus.cc:515] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.168430 6574 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc" is_pre_election: true
I20251026 08:53:58.168462 6500 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:53:58.168591 6574 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 0.
I20251026 08:53:58.168612 6500 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 0.
I20251026 08:53:58.168826 6626 leader_election.cc:290] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:53:58.168998 6390 leader_election.cc:304] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, 681cd7907543400494c8430f92a7f541; no voters:
I20251026 08:53:58.169257 6621 raft_consensus.cc:2804] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:58.169322 6621 raft_consensus.cc:493] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:58.169374 6621 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.171693 6621 raft_consensus.cc:515] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.172437 6500 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:53:58.172497 6574 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc"
I20251026 08:53:58.172588 6500 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.172618 6574 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.175170 6500 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 1.
I20251026 08:53:58.175228 6574 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 1.
I20251026 08:53:58.175450 6388 leader_election.cc:304] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681cd7907543400494c8430f92a7f541, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:53:58.175652 6621 leader_election.cc:290] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 election: Requested vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:58.175597 6499 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:53:58.175788 6499 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 0.
I20251026 08:53:58.175781 6621 raft_consensus.cc:2804] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:58.176030 6425 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541" is_pre_election: true
I20251026 08:53:58.176066 6537 leader_election.cc:304] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:53:58.176120 6621 raft_consensus.cc:697] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 LEADER]: Becoming Leader. State: Replica: 681cd7907543400494c8430f92a7f541, State: Running, Role: LEADER
I20251026 08:53:58.176163 6425 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 0.
I20251026 08:53:58.176379 6626 raft_consensus.cc:2804] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:58.176424 6626 raft_consensus.cc:493] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:58.176343 6621 consensus_queue.cc:237] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.176484 6626 raft_consensus.cc:3060] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.178324 6626 raft_consensus.cc:515] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.178706 6626 leader_election.cc:290] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 1 election: Requested vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:53:58.178539 6323 catalog_manager.cc:5649] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 681cd7907543400494c8430f92a7f541 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "681cd7907543400494c8430f92a7f541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:58.178972 6425 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541"
I20251026 08:53:58.179020 6499 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:53:58.179086 6425 raft_consensus.cc:3060] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.179107 6499 raft_consensus.cc:3060] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.180997 6425 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 1.
I20251026 08:53:58.181098 6499 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 1.
I20251026 08:53:58.181354 6538 leader_election.cc:304] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, 681cd7907543400494c8430f92a7f541; no voters:
I20251026 08:53:58.181589 6626 raft_consensus.cc:2804] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:58.181912 6626 raft_consensus.cc:697] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 LEADER]: Becoming Leader. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Running, Role: LEADER
I20251026 08:53:58.182113 6626 consensus_queue.cc:237] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.183822 6323 catalog_manager.cc:5649] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67). New cstate: current_term: 1 leader_uuid: "42b92f7783854078a49c3eee2fb0e8dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: HEALTHY } } }
I20251026 08:53:58.192293 6621 raft_consensus.cc:493] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:58.192387 6621 raft_consensus.cc:515] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.192725 6621 leader_election.cc:290] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:58.192996 6499 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:53:58.193131 6574 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc" is_pre_election: true
I20251026 08:53:58.193131 6499 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 0.
I20251026 08:53:58.193284 6574 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 0.
I20251026 08:53:58.193460 6388 leader_election.cc:304] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681cd7907543400494c8430f92a7f541, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:53:58.193761 6621 raft_consensus.cc:2804] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:58.193842 6621 raft_consensus.cc:493] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:58.193913 6621 raft_consensus.cc:3060] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.195468 6621 raft_consensus.cc:515] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.195899 6621 leader_election.cc:290] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 election: Requested vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:58.196157 6499 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:53:58.196250 6574 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc"
I20251026 08:53:58.196290 6499 raft_consensus.cc:3060] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.196363 6574 raft_consensus.cc:3060] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.198356 6574 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 1.
I20251026 08:53:58.198356 6499 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 1.
I20251026 08:53:58.198676 6390 leader_election.cc:304] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, 681cd7907543400494c8430f92a7f541; no voters:
I20251026 08:53:58.198856 6621 raft_consensus.cc:2804] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:58.198956 6621 raft_consensus.cc:697] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 LEADER]: Becoming Leader. State: Replica: 681cd7907543400494c8430f92a7f541, State: Running, Role: LEADER
I20251026 08:53:58.199121 6621 consensus_queue.cc:237] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.200748 6323 catalog_manager.cc:5649] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 reported cstate change: term changed from 0 to 1, leader changed from <none> to 681cd7907543400494c8430f92a7f541 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "681cd7907543400494c8430f92a7f541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:58.221829 6630 raft_consensus.cc:493] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:53:58.221937 6630 raft_consensus.cc:515] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.222491 6630 leader_election.cc:290] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:58.227422 6574 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc" is_pre_election: true
I20251026 08:53:58.227463 6425 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541" is_pre_election: true
I20251026 08:53:58.227550 6574 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 0.
I20251026 08:53:58.227592 6425 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 0.
I20251026 08:53:58.227838 6465 leader_election.cc:304] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:53:58.228021 6630 raft_consensus.cc:2804] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:53:58.228089 6630 raft_consensus.cc:493] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:53:58.228215 6630 raft_consensus.cc:3060] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.229826 6630 raft_consensus.cc:515] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.230377 6630 leader_election.cc:290] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 1 election: Requested vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:53:58.230713 6574 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc"
I20251026 08:53:58.230710 6425 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541"
I20251026 08:53:58.230854 6574 raft_consensus.cc:3060] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.230912 6425 raft_consensus.cc:3060] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:53:58.232308 6574 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 1.
I20251026 08:53:58.232461 6425 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 1.
I20251026 08:53:58.232561 6465 leader_election.cc:304] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:53:58.232754 6630 raft_consensus.cc:2804] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:53:58.233121 6630 raft_consensus.cc:697] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 LEADER]: Becoming Leader. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Running, Role: LEADER
I20251026 08:53:58.233343 6630 consensus_queue.cc:237] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:53:58.235582 6323 catalog_manager.cc:5649] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66). New cstate: current_term: 1 leader_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20251026 08:53:58.276510 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:53:58.284044 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:58.284353 413 tablet_replica.cc:333] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: stopping tablet replica
I20251026 08:53:58.284561 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:58.284785 413 pending_rounds.cc:70] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Trying to abort 1 pending ops.
I20251026 08:53:58.284845 413 pending_rounds.cc:77] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976361265049600 op_type: NO_OP noop_request { }
I20251026 08:53:58.284948 413 raft_consensus.cc:2889] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:58.285023 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.285771 413 tablet_replica.cc:333] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: stopping tablet replica
I20251026 08:53:58.285903 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.286028 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.286554 413 tablet_replica.cc:333] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: stopping tablet replica
I20251026 08:53:58.286679 413 raft_consensus.cc:2243] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:58.286828 413 pending_rounds.cc:70] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Trying to abort 1 pending ops.
I20251026 08:53:58.286865 413 pending_rounds.cc:77] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976361172652032 op_type: NO_OP noop_request { }
I20251026 08:53:58.286940 413 raft_consensus.cc:2889] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:58.287014 413 raft_consensus.cc:2272] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.287621 413 tablet_replica.cc:333] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: stopping tablet replica
I20251026 08:53:58.287756 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.287868 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.302130 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
I20251026 08:53:58.305279 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:53:58.312515 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:58.312803 413 tablet_replica.cc:333] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: stopping tablet replica
I20251026 08:53:58.312970 413 raft_consensus.cc:2243] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.313077 413 raft_consensus.cc:2272] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.313544 413 tablet_replica.cc:333] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: stopping tablet replica
I20251026 08:53:58.313684 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.313796 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.314275 413 tablet_replica.cc:333] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: stopping tablet replica
I20251026 08:53:58.314383 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.314512 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.314972 413 tablet_replica.cc:333] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: stopping tablet replica
I20251026 08:53:58.315099 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:58.315256 413 pending_rounds.cc:70] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Trying to abort 1 pending ops.
I20251026 08:53:58.315306 413 pending_rounds.cc:77] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976361405923328 op_type: NO_OP noop_request { }
I20251026 08:53:58.315398 413 raft_consensus.cc:2889] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:58.315492 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.329690 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
I20251026 08:53:58.332648 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:53:58.340253 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:53:58.340634 413 tablet_replica.cc:333] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: stopping tablet replica
I20251026 08:53:58.340914 413 raft_consensus.cc:2243] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.341048 413 raft_consensus.cc:2272] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.341503 413 tablet_replica.cc:333] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: stopping tablet replica
I20251026 08:53:58.341631 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:53:58.341790 413 pending_rounds.cc:70] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Trying to abort 1 pending ops.
I20251026 08:53:58.341841 413 pending_rounds.cc:77] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7214976361195614208 op_type: NO_OP noop_request { }
I20251026 08:53:58.341928 413 raft_consensus.cc:2889] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251026 08:53:58.341988 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.342577 413 tablet_replica.cc:333] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: stopping tablet replica
I20251026 08:53:58.342676 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.342773 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.343153 413 tablet_replica.cc:333] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: stopping tablet replica
I20251026 08:53:58.343243 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:53:58.343334 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:53:58.356918 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:53:58.369061 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:53:58.371171 6650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:58.371256 6651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:53:58.371635 6653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:53:58.371979 413 server_base.cc:1047] running on GCE node
I20251026 08:53:58.372162 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:53:58.372233 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:53:58.372296 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468838372292 us; error 0 us; skew 500 ppm
I20251026 08:53:58.372432 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:53:58.373488 413 webserver.cc:492] Webserver started at http://127.0.103.68:37765/ using document root <none> and password file <none>
I20251026 08:53:58.373770 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:53:58.373850 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:53:58.373993 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:53:58.374647 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "4a1e834d72814edc86de777295eac940"
format_stamp: "Formatted at 2025-10-26 08:53:58 on dist-test-slave-rd83"
I20251026 08:53:58.376856 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20251026 08:53:58.378136 6658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:58.378487 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:53:58.378597 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "4a1e834d72814edc86de777295eac940"
format_stamp: "Formatted at 2025-10-26 08:53:58 on dist-test-slave-rd83"
I20251026 08:53:58.378700 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:53:58.410356 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:53:58.410722 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:53:58.411118 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:53:58.411780 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:53:58.411832 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:58.411886 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:53:58.411924 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:53:58.423069 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:34059
I20251026 08:53:58.423100 6720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:34059 every 8 connection(s)
I20251026 08:53:58.429205 6721 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:53:58.429327 6721 heartbeater.cc:461] Registering TS with master...
I20251026 08:53:58.429625 6721 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:53:58.430408 6323 ts_manager.cc:194] Registered new tserver with Master: 4a1e834d72814edc86de777295eac940 (127.0.103.68:34059)
I20251026 08:53:58.431133 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59316
W20251026 08:53:58.820137 6371 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:53:58.821246 6307 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20251026 08:53:58.822296 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:53:58.822382 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:53:58.823809 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:53:58.823894 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:53:58.825227 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:53:58.825306 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
I20251026 08:53:59.432236 6721 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
W20251026 08:53:59.828984 6371 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:53:59.830789 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:53:59.830859 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:53:59.832115 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:53:59.832187 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:53:59.833444 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:53:59.833506 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:54:00.837680 6371 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:00.839834 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:00.839920 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:00.841234 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:54:00.841301 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:54:00.842681 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:00.842748 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:01.846712 6371 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:01.850999 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:01.851085 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:01.852452 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:54:01.852514 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.66:36383: connect: Connection refused (error 111)
W20251026 08:54:01.853907 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:01.853973 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:02.857518 6371 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:02.859341 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:02.859409 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:02.860549 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:02.860646 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41857: connect: Connection refused (error 111)
W20251026 08:54:02.861953 6371 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:02.862015 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.0.103.67:40531: connect: Connection refused (error 111)
W20251026 08:54:03.862887 6371 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 681cd7907543400494c8430f92a7f541 not available for placement
I20251026 08:54:03.944422 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:03.947364 6740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:03.947681 6742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:03.947398 6739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:03.957127 413 server_base.cc:1047] running on GCE node
I20251026 08:54:03.957487 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:03.957576 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:03.957638 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468843957619 us; error 0 us; skew 500 ppm
I20251026 08:54:03.957791 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:03.959110 413 webserver.cc:492] Webserver started at http://127.0.103.65:40537/ using document root <none> and password file <none>
I20251026 08:54:03.959338 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:03.959412 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 08:54:03.961403 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20251026 08:54:03.962884 6747 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:03.963222 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:03.963338 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "681cd7907543400494c8430f92a7f541"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:54:03.963464 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:03.973574 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:03.974049 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:03.974648 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:03.975952 6754 ts_tablet_manager.cc:548] Loading tablet metadata (0/4 complete)
I20251026 08:54:03.983971 413 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251026 08:54:03.984062 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.008s user 0.000s sys 0.000s
I20251026 08:54:03.984164 413 ts_tablet_manager.cc:600] Registering tablets (0/4 complete)
I20251026 08:54:03.986285 6754 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:54:03.989701 6754 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:03.990038 6754 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Bootstrap complete.
I20251026 08:54:03.990295 6754 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251026 08:54:03.990957 6754 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:03.991115 6754 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:54:03.991350 6754 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:03.991899 6754 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:03.992123 6754 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:54:03.992204 413 ts_tablet_manager.cc:616] Registered 4 tablets
I20251026 08:54:03.992302 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20251026 08:54:03.996107 6754 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 08:54:03.996552 6754 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Bootstrap complete.
I20251026 08:54:03.996788 6754 ts_tablet_manager.cc:1403] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251026 08:54:03.997478 6754 raft_consensus.cc:359] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:03.997694 6754 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:54:03.997996 6754 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:03.998520 6754 ts_tablet_manager.cc:1434] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 08:54:03.998744 6754 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:54:04.002521 6754 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 08:54:04.002997 6754 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Bootstrap complete.
I20251026 08:54:04.003240 6754 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:54:04.003768 6754 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.003968 6754 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.004186 6754 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.004585 6754 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:04.004804 6754 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Bootstrap starting.
I20251026 08:54:04.008025 6754 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.008387 6754 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Bootstrap complete.
I20251026 08:54:04.008600 6754 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:54:04.009429 6754 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.009621 6754 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 681cd7907543400494c8430f92a7f541, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.009931 6754 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.010324 6754 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251026 08:54:04.013195 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41857
I20251026 08:54:04.013203 6815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41857 every 8 connection(s)
I20251026 08:54:04.014511 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:04.017102 6820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:04.018209 6821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:04.020732 6823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:04.020973 413 server_base.cc:1047] running on GCE node
I20251026 08:54:04.021389 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:04.021476 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:04.021538 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468844021515 us; error 0 us; skew 500 ppm
I20251026 08:54:04.021708 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:04.022965 413 webserver.cc:492] Webserver started at http://127.0.103.66:37551/ using document root <none> and password file <none>
I20251026 08:54:04.023216 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:04.023284 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 08:54:04.025480 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20251026 08:54:04.026747 6828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:04.027037 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:04.027133 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:54:04.027240 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:04.029673 6816 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:54:04.029806 6816 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:04.030182 6816 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:54:04.031354 6323 ts_manager.cc:194] Re-registered known tserver with Master: 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857)
I20251026 08:54:04.033236 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59330
I20251026 08:54:04.043855 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:04.044239 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:04.044677 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:04.052389 413 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251026 08:54:04.052479 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20251026 08:54:04.054018 6835 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:54:04.057369 6835 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 08:54:04.057701 6835 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap complete.
I20251026 08:54:04.057891 6835 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:54:04.058306 413 ts_tablet_manager.cc:616] Registered 4 tablets
I20251026 08:54:04.058372 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20251026 08:54:04.058450 6835 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.058684 6835 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.058893 6835 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.059500 6835 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20251026 08:54:04.059744 6835 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:54:04.063603 6835 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.064013 6835 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap complete.
I20251026 08:54:04.064352 6835 ts_tablet_manager.cc:1403] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:54:04.065289 6835 raft_consensus.cc:359] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.065557 6835 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.065830 6835 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.066418 6835 ts_tablet_manager.cc:1434] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:04.066687 6835 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:54:04.070574 6835 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.070829 6835 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap complete.
I20251026 08:54:04.071028 6835 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:54:04.071619 6835 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.071755 6835 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.071853 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:36383
I20251026 08:54:04.071872 6895 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:36383 every 8 connection(s)
I20251026 08:54:04.071969 6835 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.072350 6835 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:04.072563 6835 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap starting.
I20251026 08:54:04.074328 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 08:54:04.076314 6835 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.076805 6835 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Bootstrap complete.
I20251026 08:54:04.077111 6835 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251026 08:54:04.078352 6835 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.078606 6835 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.078872 6835 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.079424 6835 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251026 08:54:04.079476 6896 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:54:04.079632 6896 heartbeater.cc:461] Registering TS with master...
W20251026 08:54:04.079680 6900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:04.080022 6896 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:54:04.081235 6323 ts_manager.cc:194] Re-registered known tserver with Master: aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
W20251026 08:54:04.081616 6901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:04.082360 6903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:04.082584 413 server_base.cc:1047] running on GCE node
I20251026 08:54:04.082815 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:04.082873 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:04.082912 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468844082891 us; error 0 us; skew 500 ppm
I20251026 08:54:04.083051 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:04.083634 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59340
I20251026 08:54:04.084053 413 webserver.cc:492] Webserver started at http://127.0.103.67:35563/ using document root <none> and password file <none>
I20251026 08:54:04.084242 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:04.084339 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 08:54:04.084709 6896 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
I20251026 08:54:04.085981 413 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20251026 08:54:04.087325 6908 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:04.087642 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:04.087721 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "42b92f7783854078a49c3eee2fb0e8dc"
format_stamp: "Formatted at 2025-10-26 08:53:56 on dist-test-slave-rd83"
I20251026 08:54:04.087800 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:04.107826 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:04.108229 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:04.108632 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:04.116454 413 ts_tablet_manager.cc:585] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20251026 08:54:04.116533 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20251026 08:54:04.118273 6915 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:54:04.121145 6915 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.121460 6915 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap complete.
I20251026 08:54:04.121696 6915 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:04.122330 6915 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.122498 6915 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.122721 6915 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.122792 413 ts_tablet_manager.cc:616] Registered 4 tablets
I20251026 08:54:04.122934 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.006s user 0.003s sys 0.003s
I20251026 08:54:04.123384 6915 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:04.123586 6915 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:54:04.127152 6915 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.127506 6915 tablet_bootstrap.cc:492] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap complete.
I20251026 08:54:04.127771 6915 ts_tablet_manager.cc:1403] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:54:04.128475 6915 raft_consensus.cc:359] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.128623 6915 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.128847 6915 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.129271 6915 ts_tablet_manager.cc:1434] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:54:04.129504 6915 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:54:04.132505 6915 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:04.132987 6915 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap complete.
I20251026 08:54:04.133288 6915 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251026 08:54:04.134171 6915 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.134409 6915 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.134702 6915 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.135274 6915 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:04.135571 6915 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap starting.
I20251026 08:54:04.137669 6975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:40531 every 8 connection(s)
I20251026 08:54:04.137676 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:40531
I20251026 08:54:04.139918 6915 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20251026 08:54:04.140427 6915 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Bootstrap complete.
I20251026 08:54:04.140759 6915 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251026 08:54:04.141366 6915 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.141618 6915 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Initialized, Role: FOLLOWER
I20251026 08:54:04.141887 6915 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:04.142275 6915 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:54:04.144742 6976 heartbeater.cc:344] Connected to a master server at 127.0.103.126:41517
I20251026 08:54:04.144909 6976 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:04.145243 6976 heartbeater.cc:507] Master 127.0.103.126:41517 requested a full tablet report, sending...
I20251026 08:54:04.146349 6323 ts_manager.cc:194] Re-registered known tserver with Master: 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:04.147962 6323 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59346
I20251026 08:54:04.148880 6976 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
W20251026 08:54:04.870971 6371 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 681cd7907543400494c8430f92a7f541 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } }
W20251026 08:54:04.872273 6371 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 43036738df554f559a3449b172705987, TS 42b92f7783854078a49c3eee2fb0e8dc -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:04.872365 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 43036738df554f559a3449b172705987, TS 42b92f7783854078a49c3eee2fb0e8dc -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:04.877946 6371 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 961f00cd332643dab9bccc5f28d9cf39, TS 681cd7907543400494c8430f92a7f541 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:04.878036 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 961f00cd332643dab9bccc5f28d9cf39, TS 681cd7907543400494c8430f92a7f541 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:04.879172 6371 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0e2c96cedcd04e5289ea308fa1f0c16f, TS aec2df26a3e6420b80dfa0a8a25af8f8 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:04.879283 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0e2c96cedcd04e5289ea308fa1f0c16f, TS aec2df26a3e6420b80dfa0a8a25af8f8 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.034511 6816 heartbeater.cc:499] Master 127.0.103.126:41517 was elected leader, sending a full tablet report...
I20251026 08:54:05.455711 6983 raft_consensus.cc:493] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:05.455879 6983 raft_consensus.cc:515] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.456574 6983 leader_election.cc:290] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:54:05.473352 6988 raft_consensus.cc:493] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:05.473585 6988 raft_consensus.cc:515] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.482211 6988 leader_election.cc:290] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:05.482838 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:54:05.483052 6870 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 1.
I20251026 08:54:05.483211 6790 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541" is_pre_election: true
I20251026 08:54:05.483455 6790 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 1.
I20251026 08:54:05.483649 6909 leader_election.cc:304] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:54:05.483948 6983 raft_consensus.cc:2804] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:54:05.484057 6983 raft_consensus.cc:493] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:05.484172 6983 raft_consensus.cc:3060] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.487003 6983 raft_consensus.cc:515] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.487550 6983 leader_election.cc:290] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 election: Requested vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:54:05.487872 6790 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541"
I20251026 08:54:05.487995 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "961f00cd332643dab9bccc5f28d9cf39" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:54:05.488025 6790 raft_consensus.cc:3060] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.488125 6870 raft_consensus.cc:3060] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.490134 6951 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc" is_pre_election: true
I20251026 08:54:05.490306 6951 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 1.
I20251026 08:54:05.490483 6789 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541" is_pre_election: true
I20251026 08:54:05.490641 6790 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 2.
I20251026 08:54:05.490705 6831 leader_election.cc:304] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:54:05.490665 6789 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 1.
I20251026 08:54:05.490641 6870 raft_consensus.cc:2468] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 2.
I20251026 08:54:05.490940 6988 raft_consensus.cc:2804] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:54:05.491030 6988 raft_consensus.cc:493] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:05.491075 6910 leader_election.cc:304] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, 681cd7907543400494c8430f92a7f541; no voters:
I20251026 08:54:05.491120 6988 raft_consensus.cc:3060] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.491402 6983 raft_consensus.cc:2804] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:54:05.491789 6983 raft_consensus.cc:697] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 LEADER]: Becoming Leader. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Running, Role: LEADER
I20251026 08:54:05.492043 6983 consensus_queue.cc:237] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.493216 6988 raft_consensus.cc:515] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.493829 6988 leader_election.cc:290] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 2 election: Requested vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:05.494199 6789 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541"
I20251026 08:54:05.494320 6951 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32f6afe780340f5912355e73ddface9" candidate_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc"
I20251026 08:54:05.494408 6789 raft_consensus.cc:3060] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.494491 6951 raft_consensus.cc:3060] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.494683 6323 catalog_manager.cc:5649] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "42b92f7783854078a49c3eee2fb0e8dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: HEALTHY } } }
I20251026 08:54:05.497133 6789 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 2.
I20251026 08:54:05.497155 6951 raft_consensus.cc:2468] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aec2df26a3e6420b80dfa0a8a25af8f8 in term 2.
I20251026 08:54:05.497527 6830 leader_election.cc:304] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 681cd7907543400494c8430f92a7f541, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:54:05.497769 6988 raft_consensus.cc:2804] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:54:05.498157 6988 raft_consensus.cc:697] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 LEADER]: Becoming Leader. State: Replica: aec2df26a3e6420b80dfa0a8a25af8f8, State: Running, Role: LEADER
I20251026 08:54:05.498611 6988 consensus_queue.cc:237] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.500789 6323 catalog_manager.cc:5649] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:05.502460 7001 raft_consensus.cc:493] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:05.502595 7001 raft_consensus.cc:515] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.503322 7001 leader_election.cc:290] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:05.508690 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:54:05.508746 6951 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc" is_pre_election: true
I20251026 08:54:05.508847 6870 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 1.
I20251026 08:54:05.508903 6951 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 1.
I20251026 08:54:05.509198 6750 leader_election.cc:304] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, 681cd7907543400494c8430f92a7f541; no voters:
I20251026 08:54:05.509419 7001 raft_consensus.cc:2804] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:54:05.509491 7001 raft_consensus.cc:493] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:05.509577 7001 raft_consensus.cc:3060] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.511195 7001 raft_consensus.cc:515] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.511596 7001 leader_election.cc:290] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 2 election: Requested vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:05.511889 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:54:05.511968 6951 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "43036738df554f559a3449b172705987" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc"
I20251026 08:54:05.512018 6870 raft_consensus.cc:3060] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.512130 6951 raft_consensus.cc:3060] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.513823 6870 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 2.
I20251026 08:54:05.514006 6951 raft_consensus.cc:2468] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 2.
I20251026 08:54:05.514354 6750 leader_election.cc:304] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, 681cd7907543400494c8430f92a7f541, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:54:05.514613 7001 raft_consensus.cc:2804] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:54:05.514945 7001 raft_consensus.cc:697] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 2 LEADER]: Becoming Leader. State: Replica: 681cd7907543400494c8430f92a7f541, State: Running, Role: LEADER
I20251026 08:54:05.515154 7001 consensus_queue.cc:237] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.517112 6323 catalog_manager.cc:5649] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "681cd7907543400494c8430f92a7f541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:05.569813 6983 raft_consensus.cc:493] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:05.569952 6983 raft_consensus.cc:515] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.570443 6983 leader_election.cc:290] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:54:05.570909 6789 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541" is_pre_election: true
I20251026 08:54:05.570932 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:54:05.571142 6870 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 1.
I20251026 08:54:05.571285 6789 raft_consensus.cc:2410] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 42b92f7783854078a49c3eee2fb0e8dc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 08:54:05.571571 6909 leader_election.cc:304] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:54:05.571830 6983 raft_consensus.cc:2804] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:54:05.571913 6983 raft_consensus.cc:493] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:05.571998 6983 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.573648 6983 raft_consensus.cc:515] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.574127 6983 leader_election.cc:290] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 election: Requested vote from peers 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857), aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:54:05.574484 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:54:05.574481 6789 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "42b92f7783854078a49c3eee2fb0e8dc" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "681cd7907543400494c8430f92a7f541"
I20251026 08:54:05.574646 6870 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.574707 6789 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:05.576530 6870 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42b92f7783854078a49c3eee2fb0e8dc in term 2.
I20251026 08:54:05.576830 6789 raft_consensus.cc:2410] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 42b92f7783854078a49c3eee2fb0e8dc for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20251026 08:54:05.576851 6909 leader_election.cc:304] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42b92f7783854078a49c3eee2fb0e8dc, aec2df26a3e6420b80dfa0a8a25af8f8; no voters:
I20251026 08:54:05.577059 6983 raft_consensus.cc:2804] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:54:05.577212 6983 raft_consensus.cc:697] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 2 LEADER]: Becoming Leader. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Running, Role: LEADER
I20251026 08:54:05.577409 6983 consensus_queue.cc:237] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.579483 6323 catalog_manager.cc:5649] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc reported cstate change: term changed from 1 to 2, leader changed from 681cd7907543400494c8430f92a7f541 (127.0.103.65) to 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67). New cstate: current_term: 2 leader_uuid: "42b92f7783854078a49c3eee2fb0e8dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: HEALTHY } } }
I20251026 08:54:05.663200 7001 raft_consensus.cc:493] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:05.663393 7001 raft_consensus.cc:515] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.663964 7001 leader_election.cc:290] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:05.664497 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" is_pre_election: true
I20251026 08:54:05.664494 6951 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc" is_pre_election: true
I20251026 08:54:05.664716 6870 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 2.
I20251026 08:54:05.665058 6750 leader_election.cc:304] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681cd7907543400494c8430f92a7f541, aec2df26a3e6420b80dfa0a8a25af8f8; no voters: 42b92f7783854078a49c3eee2fb0e8dc
I20251026 08:54:05.665362 7001 raft_consensus.cc:2804] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251026 08:54:05.665453 7001 raft_consensus.cc:493] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:05.665534 7001 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:54:05.667356 7001 raft_consensus.cc:515] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.667797 7001 leader_election.cc:290] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 3 election: Requested vote from peers aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383), 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:05.668149 6870 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8"
I20251026 08:54:05.668195 6951 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0e2c96cedcd04e5289ea308fa1f0c16f" candidate_uuid: "681cd7907543400494c8430f92a7f541" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "42b92f7783854078a49c3eee2fb0e8dc"
I20251026 08:54:05.668296 6870 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:54:05.670007 6870 raft_consensus.cc:2468] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 681cd7907543400494c8430f92a7f541 in term 3.
I20251026 08:54:05.670346 6748 leader_election.cc:304] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 681cd7907543400494c8430f92a7f541, aec2df26a3e6420b80dfa0a8a25af8f8; no voters: 42b92f7783854078a49c3eee2fb0e8dc
I20251026 08:54:05.670553 7001 raft_consensus.cc:2804] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 3 FOLLOWER]: Leader election won for term 3
I20251026 08:54:05.670667 7001 raft_consensus.cc:697] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 3 LEADER]: Becoming Leader. State: Replica: 681cd7907543400494c8430f92a7f541, State: Running, Role: LEADER
I20251026 08:54:05.670938 7001 consensus_queue.cc:237] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.673003 6323 catalog_manager.cc:5649] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 reported cstate change: term changed from 2 to 3, leader changed from 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67) to 681cd7907543400494c8430f92a7f541 (127.0.103.65). New cstate: current_term: 3 leader_uuid: "681cd7907543400494c8430f92a7f541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: UNKNOWN } } }
W20251026 08:54:05.887355 6371 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20251026 08:54:05.888861 6371 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet d32f6afe780340f5912355e73ddface9, TS 42b92f7783854078a49c3eee2fb0e8dc -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:05.888936 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet d32f6afe780340f5912355e73ddface9, TS 42b92f7783854078a49c3eee2fb0e8dc -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:05.890247 6371 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 43036738df554f559a3449b172705987, TS aec2df26a3e6420b80dfa0a8a25af8f8 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:05.890324 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 43036738df554f559a3449b172705987, TS aec2df26a3e6420b80dfa0a8a25af8f8 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:05.904091 6371 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 961f00cd332643dab9bccc5f28d9cf39, TS 681cd7907543400494c8430f92a7f541 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
W20251026 08:54:05.904153 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 961f00cd332643dab9bccc5f28d9cf39, TS 681cd7907543400494c8430f92a7f541 -> TS 4a1e834d72814edc86de777295eac940 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:05.923790 6870 raft_consensus.cc:1275] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Refusing update from remote peer 42b92f7783854078a49c3eee2fb0e8dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:05.924221 6983 consensus_queue.cc:1048] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:05.928241 6789 raft_consensus.cc:1275] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Refusing update from remote peer 42b92f7783854078a49c3eee2fb0e8dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:05.928751 6983 consensus_queue.cc:1048] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:05.945600 6951 raft_consensus.cc:1275] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Refusing update from remote peer 681cd7907543400494c8430f92a7f541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:05.946054 7001 consensus_queue.cc:1048] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:05.949292 6870 raft_consensus.cc:1275] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Refusing update from remote peer 681cd7907543400494c8430f92a7f541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:05.949752 7016 consensus_queue.cc:1048] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:05.994984 6951 raft_consensus.cc:1275] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Refusing update from remote peer aec2df26a3e6420b80dfa0a8a25af8f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:05.995443 6999 consensus_queue.cc:1048] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:05.999339 6789 raft_consensus.cc:1275] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Refusing update from remote peer aec2df26a3e6420b80dfa0a8a25af8f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:05.999697 7019 consensus_queue.cc:1048] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:06.083493 6951 raft_consensus.cc:3055] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 2 LEADER]: Stepping down as leader of term 2
I20251026 08:54:06.083580 6951 raft_consensus.cc:740] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 42b92f7783854078a49c3eee2fb0e8dc, State: Running, Role: LEADER
I20251026 08:54:06.083774 6951 consensus_queue.cc:260] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } }
I20251026 08:54:06.083982 6951 raft_consensus.cc:3060] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Advancing to term 3
I20251026 08:54:06.085912 6951 raft_consensus.cc:1275] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 3 FOLLOWER]: Refusing update from remote peer 681cd7907543400494c8430f92a7f541: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 08:54:06.086373 7016 consensus_queue.cc:1048] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:06.086560 7016 consensus_queue.cc:1243] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [LEADER]: Peer 42b92f7783854078a49c3eee2fb0e8dc log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20251026 08:54:06.087392 6951 pending_rounds.cc:85] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Aborting all ops after (but not including) 0
I20251026 08:54:06.087466 6951 pending_rounds.cc:107] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20251026 08:54:06.087536 6951 raft_consensus.cc:2889] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20251026 08:54:06.089604 6870 raft_consensus.cc:1275] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 3 FOLLOWER]: Refusing update from remote peer 681cd7907543400494c8430f92a7f541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20251026 08:54:06.090112 7016 consensus_queue.cc:1048] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:06.907454 6789 consensus_queue.cc:237] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:06.909147 6870 raft_consensus.cc:1275] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Refusing update from remote peer 681cd7907543400494c8430f92a7f541: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:54:06.909706 6951 raft_consensus.cc:1275] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Refusing update from remote peer 681cd7907543400494c8430f92a7f541: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:54:06.909729 7004 consensus_queue.cc:1048] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:06.910255 7001 consensus_queue.cc:1048] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:06.912225 7001 raft_consensus.cc:2955] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.912575 6870 raft_consensus.cc:2955] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.912674 6951 raft_consensus.cc:2955] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
W20251026 08:54:06.917200 6749 consensus_peers.cc:597] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 -> Peer 4a1e834d72814edc86de777295eac940 (127.0.103.68:34059): Couldn't send request to peer 4a1e834d72814edc86de777295eac940. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43036738df554f559a3449b172705987. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:06.917506 6322 catalog_manager.cc:5649] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 reported cstate change: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "681cd7907543400494c8430f92a7f541" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.920483 6870 consensus_queue.cc:237] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:06.922941 6951 raft_consensus.cc:1275] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Refusing update from remote peer aec2df26a3e6420b80dfa0a8a25af8f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:54:06.923504 6999 consensus_queue.cc:1048] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:06.923878 6789 raft_consensus.cc:1275] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Refusing update from remote peer aec2df26a3e6420b80dfa0a8a25af8f8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:54:06.924500 7022 consensus_queue.cc:1048] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:06.925695 6999 raft_consensus.cc:2955] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.927893 6951 raft_consensus.cc:2955] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.928725 6789 raft_consensus.cc:2955] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.930939 6949 consensus_queue.cc:237] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
W20251026 08:54:06.932432 6830 consensus_peers.cc:597] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 -> Peer 4a1e834d72814edc86de777295eac940 (127.0.103.68:34059): Couldn't send request to peer 4a1e834d72814edc86de777295eac940. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d32f6afe780340f5912355e73ddface9. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:06.934196 6870 raft_consensus.cc:1275] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Refusing update from remote peer 42b92f7783854078a49c3eee2fb0e8dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:54:06.934020 6323 catalog_manager.cc:5649] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 reported cstate change: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.934669 7011 consensus_queue.cc:1048] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:06.934731 6789 raft_consensus.cc:1275] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Refusing update from remote peer 42b92f7783854078a49c3eee2fb0e8dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251026 08:54:06.935285 7011 consensus_queue.cc:1048] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:06.937134 7032 raft_consensus.cc:2955] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.938623 6870 raft_consensus.cc:2955] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:06.941404 6322 catalog_manager.cc:5649] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc reported cstate change: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New cstate: current_term: 2 leader_uuid: "42b92f7783854078a49c3eee2fb0e8dc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20251026 08:54:06.944150 6910 consensus_peers.cc:597] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc -> Peer 4a1e834d72814edc86de777295eac940 (127.0.103.68:34059): Couldn't send request to peer 4a1e834d72814edc86de777295eac940. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 961f00cd332643dab9bccc5f28d9cf39. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:06.944653 6789 raft_consensus.cc:2955] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4a1e834d72814edc86de777295eac940 (127.0.103.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } } }
I20251026 08:54:07.408529 7036 ts_tablet_manager.cc:933] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: Initiating tablet copy from peer 681cd7907543400494c8430f92a7f541 (127.0.103.65:41857)
I20251026 08:54:07.409070 7036 tablet_copy_client.cc:323] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.65:41857
I20251026 08:54:07.414789 6801 tablet_copy_service.cc:140] P 681cd7907543400494c8430f92a7f541: Received BeginTabletCopySession request for tablet 43036738df554f559a3449b172705987 from peer 4a1e834d72814edc86de777295eac940 ({username='slave'} at 127.0.0.1:60086)
I20251026 08:54:07.414903 6801 tablet_copy_service.cc:161] P 681cd7907543400494c8430f92a7f541: Beginning new tablet copy session on tablet 43036738df554f559a3449b172705987 from peer 4a1e834d72814edc86de777295eac940 at {username='slave'} at 127.0.0.1:60086: session id = 4a1e834d72814edc86de777295eac940-43036738df554f559a3449b172705987
I20251026 08:54:07.417165 6801 tablet_copy_source_session.cc:215] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:07.418197 7036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43036738df554f559a3449b172705987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:07.423353 7036 tablet_copy_client.cc:806] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:07.423565 7036 tablet_copy_client.cc:670] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:07.424871 7036 tablet_copy_client.cc:538] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:07.427809 7036 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: Bootstrap starting.
I20251026 08:54:07.430178 7040 ts_tablet_manager.cc:933] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: Initiating tablet copy from peer aec2df26a3e6420b80dfa0a8a25af8f8 (127.0.103.66:36383)
I20251026 08:54:07.431455 7040 tablet_copy_client.cc:323] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:36383
I20251026 08:54:07.433431 7036 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:07.433707 7036 tablet_bootstrap.cc:492] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: Bootstrap complete.
I20251026 08:54:07.433912 7036 ts_tablet_manager.cc:1403] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.001s
I20251026 08:54:07.434819 7036 raft_consensus.cc:359] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:07.435092 7036 raft_consensus.cc:740] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4a1e834d72814edc86de777295eac940, State: Initialized, Role: LEARNER
I20251026 08:54:07.435344 7036 consensus_queue.cc:260] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } attrs { replace: true } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:07.436502 7036 ts_tablet_manager.cc:1434] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:54:07.437100 6801 tablet_copy_service.cc:342] P 681cd7907543400494c8430f92a7f541: Request end of tablet copy session 4a1e834d72814edc86de777295eac940-43036738df554f559a3449b172705987 received from {username='slave'} at 127.0.0.1:60086
I20251026 08:54:07.437191 6801 tablet_copy_service.cc:434] P 681cd7907543400494c8430f92a7f541: ending tablet copy session 4a1e834d72814edc86de777295eac940-43036738df554f559a3449b172705987 on tablet 43036738df554f559a3449b172705987 with peer 4a1e834d72814edc86de777295eac940
I20251026 08:54:07.438534 6880 tablet_copy_service.cc:140] P aec2df26a3e6420b80dfa0a8a25af8f8: Received BeginTabletCopySession request for tablet d32f6afe780340f5912355e73ddface9 from peer 4a1e834d72814edc86de777295eac940 ({username='slave'} at 127.0.0.1:38074)
I20251026 08:54:07.438635 6880 tablet_copy_service.cc:161] P aec2df26a3e6420b80dfa0a8a25af8f8: Beginning new tablet copy session on tablet d32f6afe780340f5912355e73ddface9 from peer 4a1e834d72814edc86de777295eac940 at {username='slave'} at 127.0.0.1:38074: session id = 4a1e834d72814edc86de777295eac940-d32f6afe780340f5912355e73ddface9
I20251026 08:54:07.441001 6880 tablet_copy_source_session.cc:215] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:07.442466 7040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32f6afe780340f5912355e73ddface9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:07.447189 7040 tablet_copy_client.cc:806] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:07.447440 7040 tablet_copy_client.cc:670] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:07.448539 7040 tablet_copy_client.cc:538] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:07.451006 7040 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: Bootstrap starting.
I20251026 08:54:07.455684 7040 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:07.456027 7040 tablet_bootstrap.cc:492] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: Bootstrap complete.
I20251026 08:54:07.456203 7040 ts_tablet_manager.cc:1403] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251026 08:54:07.456817 7040 raft_consensus.cc:359] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:07.456979 7040 raft_consensus.cc:740] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4a1e834d72814edc86de777295eac940, State: Initialized, Role: LEARNER
I20251026 08:54:07.457140 7040 consensus_queue.cc:260] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } attrs { replace: true } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:07.457679 7040 ts_tablet_manager.cc:1434] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:54:07.458310 6880 tablet_copy_service.cc:342] P aec2df26a3e6420b80dfa0a8a25af8f8: Request end of tablet copy session 4a1e834d72814edc86de777295eac940-d32f6afe780340f5912355e73ddface9 received from {username='slave'} at 127.0.0.1:38074
I20251026 08:54:07.458405 6880 tablet_copy_service.cc:434] P aec2df26a3e6420b80dfa0a8a25af8f8: ending tablet copy session 4a1e834d72814edc86de777295eac940-d32f6afe780340f5912355e73ddface9 on tablet d32f6afe780340f5912355e73ddface9 with peer 4a1e834d72814edc86de777295eac940
I20251026 08:54:07.468463 7040 ts_tablet_manager.cc:933] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: Initiating tablet copy from peer 42b92f7783854078a49c3eee2fb0e8dc (127.0.103.67:40531)
I20251026 08:54:07.468875 7040 tablet_copy_client.cc:323] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:40531
I20251026 08:54:07.474768 6961 tablet_copy_service.cc:140] P 42b92f7783854078a49c3eee2fb0e8dc: Received BeginTabletCopySession request for tablet 961f00cd332643dab9bccc5f28d9cf39 from peer 4a1e834d72814edc86de777295eac940 ({username='slave'} at 127.0.0.1:58326)
I20251026 08:54:07.474871 6961 tablet_copy_service.cc:161] P 42b92f7783854078a49c3eee2fb0e8dc: Beginning new tablet copy session on tablet 961f00cd332643dab9bccc5f28d9cf39 from peer 4a1e834d72814edc86de777295eac940 at {username='slave'} at 127.0.0.1:58326: session id = 4a1e834d72814edc86de777295eac940-961f00cd332643dab9bccc5f28d9cf39
I20251026 08:54:07.476931 6961 tablet_copy_source_session.cc:215] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:07.477692 7040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961f00cd332643dab9bccc5f28d9cf39. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:07.481406 7040 tablet_copy_client.cc:806] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:07.481613 7040 tablet_copy_client.cc:670] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:07.482805 7040 tablet_copy_client.cc:538] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:07.484834 7040 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: Bootstrap starting.
I20251026 08:54:07.489717 7040 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:07.490036 7040 tablet_bootstrap.cc:492] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: Bootstrap complete.
I20251026 08:54:07.490242 7040 ts_tablet_manager.cc:1403] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.006s
I20251026 08:54:07.490715 7040 raft_consensus.cc:359] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:07.490849 7040 raft_consensus.cc:740] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4a1e834d72814edc86de777295eac940, State: Initialized, Role: LEARNER
I20251026 08:54:07.491017 7040 consensus_queue.cc:260] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "681cd7907543400494c8430f92a7f541" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41857 } attrs { replace: true } } peers { permanent_uuid: "aec2df26a3e6420b80dfa0a8a25af8f8" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 36383 } } peers { permanent_uuid: "42b92f7783854078a49c3eee2fb0e8dc" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 40531 } } peers { permanent_uuid: "4a1e834d72814edc86de777295eac940" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 34059 } attrs { promote: true } }
I20251026 08:54:07.491397 7040 ts_tablet_manager.cc:1434] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 08:54:07.491961 6961 tablet_copy_service.cc:342] P 42b92f7783854078a49c3eee2fb0e8dc: Request end of tablet copy session 4a1e834d72814edc86de777295eac940-961f00cd332643dab9bccc5f28d9cf39 received from {username='slave'} at 127.0.0.1:58326
I20251026 08:54:07.492054 6961 tablet_copy_service.cc:434] P 42b92f7783854078a49c3eee2fb0e8dc: ending tablet copy session 4a1e834d72814edc86de777295eac940-961f00cd332643dab9bccc5f28d9cf39 on tablet 961f00cd332643dab9bccc5f28d9cf39 with peer 4a1e834d72814edc86de777295eac940
I20251026 08:54:07.917030 6696 raft_consensus.cc:1217] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251026 08:54:07.961058 6696 raft_consensus.cc:1217] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251026 08:54:08.002323 6696 raft_consensus.cc:1217] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20251026 08:54:08.165391 413 tablet_server.cc:178] TabletServer@127.0.103.65:41857 shutting down...
W20251026 08:54:08.167729 6368 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 14 similar messages]
W20251026 08:54:08.169214 6371 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:54:08.169332 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:54:08.173391 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:08.173704 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.173920 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:54:08.174333 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.175016 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.175169 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.175364 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.175899 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.176030 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.176174 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P 681cd7907543400494c8430f92a7f541 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.176644 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.176761 413 raft_consensus.cc:2243] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 3 LEADER]: Raft consensus shutting down.
I20251026 08:54:08.177006 413 raft_consensus.cc:2272] T 0e2c96cedcd04e5289ea308fa1f0c16f P 681cd7907543400494c8430f92a7f541 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.191948 413 tablet_server.cc:195] TabletServer@127.0.103.65:41857 shutdown complete.
I20251026 08:54:08.195645 413 tablet_server.cc:178] TabletServer@127.0.103.66:36383 shutting down...
W20251026 08:54:08.198501 6371 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:54:08.198580 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:54:08.204977 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:08.205178 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.205361 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.205574 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.206224 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.206358 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:54:08.206660 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.207105 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.207219 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.207406 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P aec2df26a3e6420b80dfa0a8a25af8f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.207789 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.207907 413 raft_consensus.cc:2243] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.208053 413 raft_consensus.cc:2272] T 0e2c96cedcd04e5289ea308fa1f0c16f P aec2df26a3e6420b80dfa0a8a25af8f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.213039 413 tablet_server.cc:195] TabletServer@127.0.103.66:36383 shutdown complete.
I20251026 08:54:08.224622 413 tablet_server.cc:178] TabletServer@127.0.103.67:40531 shutting down...
W20251026 08:54:08.227345 6371 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20251026 08:54:08.227429 6371 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20251026 08:54:08.231799 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:08.231968 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.232148 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.232306 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.232707 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.232820 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.232951 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.233291 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.233398 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:54:08.233650 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P 42b92f7783854078a49c3eee2fb0e8dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.233995 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:08.234115 413 raft_consensus.cc:2243] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 3 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:08.234223 413 raft_consensus.cc:2272] T 0e2c96cedcd04e5289ea308fa1f0c16f P 42b92f7783854078a49c3eee2fb0e8dc [term 3 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.247609 413 tablet_server.cc:195] TabletServer@127.0.103.67:40531 shutdown complete.
I20251026 08:54:08.250461 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:54:08.257459 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:08.257630 413 tablet_replica.cc:333] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940: stopping tablet replica
I20251026 08:54:08.257786 413 raft_consensus.cc:2243] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Raft consensus shutting down.
I20251026 08:54:08.257898 413 raft_consensus.cc:2272] T 961f00cd332643dab9bccc5f28d9cf39 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Raft consensus is shut down!
I20251026 08:54:08.258333 413 tablet_replica.cc:333] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940: stopping tablet replica
I20251026 08:54:08.258447 413 raft_consensus.cc:2243] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Raft consensus shutting down.
I20251026 08:54:08.258549 413 raft_consensus.cc:2272] T 43036738df554f559a3449b172705987 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Raft consensus is shut down!
I20251026 08:54:08.258901 413 tablet_replica.cc:333] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940: stopping tablet replica
I20251026 08:54:08.259009 413 raft_consensus.cc:2243] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Raft consensus shutting down.
I20251026 08:54:08.259107 413 raft_consensus.cc:2272] T d32f6afe780340f5912355e73ddface9 P 4a1e834d72814edc86de777295eac940 [term 2 LEARNER]: Raft consensus is shut down!
I20251026 08:54:08.271929 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:54:08.274648 413 master.cc:561] Master@127.0.103.126:41517 shutting down...
I20251026 08:54:08.281618 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:08.281772 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:08.281881 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 47abce3dc38645ebb6ec5b4fcc3438b7: stopping tablet replica
I20251026 08:54:08.294703 413 master.cc:583] Master@127.0.103.126:41517 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (11551 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20251026 08:54:08.309444 413 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.103.126:36111
I20251026 08:54:08.309892 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:08.311928 7056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.312165 7057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.312477 7059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:08.318434 413 server_base.cc:1047] running on GCE node
I20251026 08:54:08.318670 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:08.318719 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:08.318781 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468848318752 us; error 0 us; skew 500 ppm
I20251026 08:54:08.318917 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:08.319967 413 webserver.cc:492] Webserver started at http://127.0.103.126:33943/ using document root <none> and password file <none>
I20251026 08:54:08.320163 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:08.320232 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:54:08.320359 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:54:08.320956 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/master-0-root/instance:
uuid: "546cf24eda40447097b89835df9094f8"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.323298 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:54:08.324612 7064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.324987 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:08.325062 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/master-0-root
uuid: "546cf24eda40447097b89835df9094f8"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.325150 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:08.334348 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:08.334693 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:08.343713 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.126:36111
I20251026 08:54:08.343736 7115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.126:36111 every 8 connection(s)
I20251026 08:54:08.344828 7116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:08.348486 7116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Bootstrap starting.
I20251026 08:54:08.349458 7116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:08.350720 7116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: No bootstrap required, opened a new log
I20251026 08:54:08.351395 7116 raft_consensus.cc:359] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546cf24eda40447097b89835df9094f8" member_type: VOTER }
I20251026 08:54:08.351508 7116 raft_consensus.cc:385] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:08.351569 7116 raft_consensus.cc:740] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 546cf24eda40447097b89835df9094f8, State: Initialized, Role: FOLLOWER
I20251026 08:54:08.351797 7116 consensus_queue.cc:260] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546cf24eda40447097b89835df9094f8" member_type: VOTER }
I20251026 08:54:08.351912 7116 raft_consensus.cc:399] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251026 08:54:08.351980 7116 raft_consensus.cc:493] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251026 08:54:08.352063 7116 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:08.353634 7116 raft_consensus.cc:515] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546cf24eda40447097b89835df9094f8" member_type: VOTER }
I20251026 08:54:08.353819 7116 leader_election.cc:304] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 546cf24eda40447097b89835df9094f8; no voters:
I20251026 08:54:08.354185 7116 leader_election.cc:290] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251026 08:54:08.354257 7119 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:08.354650 7119 raft_consensus.cc:697] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 1 LEADER]: Becoming Leader. State: Replica: 546cf24eda40447097b89835df9094f8, State: Running, Role: LEADER
I20251026 08:54:08.354909 7119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546cf24eda40447097b89835df9094f8" member_type: VOTER }
I20251026 08:54:08.355027 7116 sys_catalog.cc:565] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [sys.catalog]: configured and running, proceeding with master startup.
I20251026 08:54:08.355559 7121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 546cf24eda40447097b89835df9094f8. Latest consensus state: current_term: 1 leader_uuid: "546cf24eda40447097b89835df9094f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546cf24eda40447097b89835df9094f8" member_type: VOTER } }
I20251026 08:54:08.355523 7120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "546cf24eda40447097b89835df9094f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "546cf24eda40447097b89835df9094f8" member_type: VOTER } }
I20251026 08:54:08.355685 7121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [sys.catalog]: This master's current role is: LEADER
I20251026 08:54:08.355705 7120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [sys.catalog]: This master's current role is: LEADER
I20251026 08:54:08.356640 7123 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251026 08:54:08.357604 7123 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251026 08:54:08.360543 7123 catalog_manager.cc:1357] Generated new cluster ID: 120b64cfc38a45899d32a73f24dab5ad
I20251026 08:54:08.360618 7123 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251026 08:54:08.361449 413 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20251026 08:54:08.366631 7123 catalog_manager.cc:1380] Generated new certificate authority record
I20251026 08:54:08.367347 7123 catalog_manager.cc:1514] Loading token signing keys...
I20251026 08:54:08.374958 7123 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Generated new TSK 0
I20251026 08:54:08.375149 7123 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251026 08:54:08.377584 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:08.379781 7137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.379925 7138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.380106 7140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:08.397140 413 server_base.cc:1047] running on GCE node
I20251026 08:54:08.398859 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:08.398913 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:08.398945 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468848398942 us; error 0 us; skew 500 ppm
I20251026 08:54:08.399066 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:08.400029 413 webserver.cc:492] Webserver started at http://127.0.103.65:36929/ using document root <none> and password file <none>
I20251026 08:54:08.400228 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:08.400287 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:54:08.400400 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:54:08.400969 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/instance:
uuid: "16a1439266f547fcb516c394c7025029"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.403123 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:54:08.404393 7145 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.404650 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:08.404753 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "16a1439266f547fcb516c394c7025029"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.404865 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:08.418051 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:08.418524 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:08.418897 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:08.419533 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:54:08.419600 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.419653 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:54:08.419693 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.430178 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41911
I20251026 08:54:08.430188 7207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41911 every 8 connection(s)
I20251026 08:54:08.431538 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:08.433910 7212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.434336 7213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.434824 7215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:08.435920 413 server_base.cc:1047] running on GCE node
I20251026 08:54:08.436113 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:08.436174 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:08.436249 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468848436241 us; error 0 us; skew 500 ppm
I20251026 08:54:08.436676 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:08.437754 413 webserver.cc:492] Webserver started at http://127.0.103.66:35627/ using document root <none> and password file <none>
I20251026 08:54:08.437929 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:08.437986 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:54:08.437999 7208 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36111
I20251026 08:54:08.438144 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:54:08.438150 7208 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:08.438421 7208 heartbeater.cc:507] Master 127.0.103.126:36111 requested a full tablet report, sending...
I20251026 08:54:08.438661 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/instance:
uuid: "618784080c4148edb8e4a00574b8d60a"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.439210 7081 ts_manager.cc:194] Registered new tserver with Master: 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911)
I20251026 08:54:08.440436 7081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39212
I20251026 08:54:08.441231 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:54:08.442561 7220 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.442879 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251026 08:54:08.442947 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root
uuid: "618784080c4148edb8e4a00574b8d60a"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.443013 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:08.456670 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:08.457017 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:08.457392 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:08.466102 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:54:08.466156 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.466216 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:54:08.466255 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.475777 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.66:41511
I20251026 08:54:08.475871 7282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.66:41511 every 8 connection(s)
I20251026 08:54:08.477051 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:08.479084 7286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:08.479837 7287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:08.482156 413 server_base.cc:1047] running on GCE node
W20251026 08:54:08.482697 7289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:08.482976 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20251026 08:54:08.483042 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:08.483068 7283 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36111
I20251026 08:54:08.483083 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468848483079 us; error 0 us; skew 500 ppm
I20251026 08:54:08.483189 7283 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:08.483232 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:08.483429 7283 heartbeater.cc:507] Master 127.0.103.126:36111 requested a full tablet report, sending...
I20251026 08:54:08.484138 7081 ts_manager.cc:194] Registered new tserver with Master: 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:08.484208 413 webserver.cc:492] Webserver started at http://127.0.103.67:33869/ using document root <none> and password file <none>
I20251026 08:54:08.484369 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:08.484426 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:54:08.484519 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:54:08.484825 7081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39220
I20251026 08:54:08.485057 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root/instance:
uuid: "b69381bade8d49e8802175e17fa72440"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.487154 413 fs_manager.cc:696] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20251026 08:54:08.488274 7294 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.488516 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:08.488595 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root
uuid: "b69381bade8d49e8802175e17fa72440"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:08.488670 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:08.509073 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:08.509426 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:08.509824 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:08.510470 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:54:08.510530 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.510587 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:54:08.510622 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:08.521376 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.67:44829
I20251026 08:54:08.521409 7356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.67:44829 every 8 connection(s)
I20251026 08:54:08.527082 7357 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36111
I20251026 08:54:08.527173 7357 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:08.527412 7357 heartbeater.cc:507] Master 127.0.103.126:36111 requested a full tablet report, sending...
I20251026 08:54:08.528052 7081 ts_manager.cc:194] Registered new tserver with Master: b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:08.528678 7081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39222
I20251026 08:54:08.528891 413 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.006693144s
I20251026 08:54:09.441756 7208 heartbeater.cc:499] Master 127.0.103.126:36111 was elected leader, sending a full tablet report...
I20251026 08:54:09.485873 7283 heartbeater.cc:499] Master 127.0.103.126:36111 was elected leader, sending a full tablet report...
I20251026 08:54:09.529876 7357 heartbeater.cc:499] Master 127.0.103.126:36111 was elected leader, sending a full tablet report...
I20251026 08:54:09.555859 413 test_util.cc:276] Using random seed: -317748852
I20251026 08:54:09.570232 7081 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39232:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:54:09.570914 7081 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:54:09.589322 7322 tablet_service.cc:1505] Processing CreateTablet for tablet 86e6554eb93645fe95124508aa0a4171 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:54:09.589725 7322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e6554eb93645fe95124508aa0a4171. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.589668 7321 tablet_service.cc:1505] Processing CreateTablet for tablet 95d18ab1a92641e3bde0717dcc998271 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:54:09.590407 7320 tablet_service.cc:1505] Processing CreateTablet for tablet a7b072f91e494d8b995f146c2a9fdc53 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:54:09.591068 7319 tablet_service.cc:1505] Processing CreateTablet for tablet 8f780788b372466bbc705743cd502bba (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:54:09.591524 7321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18ab1a92641e3bde0717dcc998271. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.594487 7173 tablet_service.cc:1505] Processing CreateTablet for tablet 86e6554eb93645fe95124508aa0a4171 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:54:09.594794 7172 tablet_service.cc:1505] Processing CreateTablet for tablet 95d18ab1a92641e3bde0717dcc998271 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:54:09.594944 7173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e6554eb93645fe95124508aa0a4171. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.594868 7170 tablet_service.cc:1505] Processing CreateTablet for tablet 8f780788b372466bbc705743cd502bba (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:54:09.595230 7172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18ab1a92641e3bde0717dcc998271. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.595738 7248 tablet_service.cc:1505] Processing CreateTablet for tablet 86e6554eb93645fe95124508aa0a4171 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:54:09.596132 7248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86e6554eb93645fe95124508aa0a4171. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.596212 7247 tablet_service.cc:1505] Processing CreateTablet for tablet 95d18ab1a92641e3bde0717dcc998271 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:54:09.596567 7247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18ab1a92641e3bde0717dcc998271. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.602120 7245 tablet_service.cc:1505] Processing CreateTablet for tablet 8f780788b372466bbc705743cd502bba (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:54:09.602456 7246 tablet_service.cc:1505] Processing CreateTablet for tablet a7b072f91e494d8b995f146c2a9fdc53 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:54:09.602602 7245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f780788b372466bbc705743cd502bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.602496 7171 tablet_service.cc:1505] Processing CreateTablet for tablet a7b072f91e494d8b995f146c2a9fdc53 (DEFAULT_TABLE table=test-workload [id=37afc5f2418743bb9dbae227c4793b5f]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:54:09.602840 7246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b072f91e494d8b995f146c2a9fdc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.606288 7319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f780788b372466bbc705743cd502bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.608325 7320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b072f91e494d8b995f146c2a9fdc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.610656 7171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b072f91e494d8b995f146c2a9fdc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.614404 7170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f780788b372466bbc705743cd502bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.633793 7378 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.635658 7378 tablet_bootstrap.cc:654] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.638100 7379 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.640023 7379 tablet_bootstrap.cc:654] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.641741 7377 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.642007 7379 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.642143 7379 ts_tablet_manager.cc:1403] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20251026 08:54:09.642686 7379 raft_consensus.cc:359] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.642827 7379 raft_consensus.cc:385] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.642892 7379 raft_consensus.cc:740] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.643087 7379 consensus_queue.cc:260] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.643419 7378 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.643415 7377 tablet_bootstrap.cc:654] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.643608 7378 ts_tablet_manager.cc:1403] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20251026 08:54:09.644080 7379 ts_tablet_manager.cc:1434] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:54:09.644342 7379 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.644456 7378 raft_consensus.cc:359] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.644742 7378 raft_consensus.cc:385] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.644949 7378 raft_consensus.cc:740] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.645422 7378 consensus_queue.cc:260] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.645771 7379 tablet_bootstrap.cc:654] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.650434 7378 ts_tablet_manager.cc:1434] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.007s user 0.001s sys 0.005s
I20251026 08:54:09.651197 7378 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.651225 7377 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.651396 7377 ts_tablet_manager.cc:1403] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20251026 08:54:09.652058 7377 raft_consensus.cc:359] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.652232 7377 raft_consensus.cc:385] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.652302 7377 raft_consensus.cc:740] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.652488 7378 tablet_bootstrap.cc:654] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.652514 7377 consensus_queue.cc:260] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.653573 7377 ts_tablet_manager.cc:1434] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:54:09.653956 7377 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.655951 7377 tablet_bootstrap.cc:654] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.657446 7379 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.657552 7379 ts_tablet_manager.cc:1403] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20251026 08:54:09.657799 7377 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.657900 7377 ts_tablet_manager.cc:1403] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251026 08:54:09.658221 7379 raft_consensus.cc:359] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.658366 7379 raft_consensus.cc:385] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.658452 7379 raft_consensus.cc:740] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.658512 7377 raft_consensus.cc:359] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.658646 7377 raft_consensus.cc:385] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.658744 7377 raft_consensus.cc:740] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.658633 7379 consensus_queue.cc:260] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.658936 7377 consensus_queue.cc:260] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.659042 7378 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.659184 7379 ts_tablet_manager.cc:1434] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:09.659272 7378 ts_tablet_manager.cc:1403] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.008s
I20251026 08:54:09.659330 7377 ts_tablet_manager.cc:1434] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20251026 08:54:09.659386 7379 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.659932 7377 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.659986 7378 raft_consensus.cc:359] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.660240 7378 raft_consensus.cc:385] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.660355 7378 raft_consensus.cc:740] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.660753 7379 tablet_bootstrap.cc:654] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.660632 7378 consensus_queue.cc:260] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.661697 7377 tablet_bootstrap.cc:654] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.664625 7378 ts_tablet_manager.cc:1434] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251026 08:54:09.665654 7378 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.666158 7377 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.666345 7377 ts_tablet_manager.cc:1403] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.005s
I20251026 08:54:09.667095 7378 tablet_bootstrap.cc:654] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.667284 7377 raft_consensus.cc:359] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.667498 7377 raft_consensus.cc:385] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.667621 7377 raft_consensus.cc:740] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.667902 7377 consensus_queue.cc:260] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.668242 7377 ts_tablet_manager.cc:1434] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:54:09.668485 7377 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.668521 7378 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.668675 7378 ts_tablet_manager.cc:1403] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20251026 08:54:09.669397 7377 tablet_bootstrap.cc:654] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.669353 7378 raft_consensus.cc:359] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.669545 7378 raft_consensus.cc:385] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.669603 7378 raft_consensus.cc:740] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.669795 7378 consensus_queue.cc:260] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.670718 7378 ts_tablet_manager.cc:1434] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:54:09.670837 7377 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.670987 7377 ts_tablet_manager.cc:1403] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20251026 08:54:09.671041 7378 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.671775 7377 raft_consensus.cc:359] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.671952 7377 raft_consensus.cc:385] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.672039 7377 raft_consensus.cc:740] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.672281 7386 raft_consensus.cc:493] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:09.672240 7377 consensus_queue.cc:260] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.672375 7386 raft_consensus.cc:515] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.672521 7378 tablet_bootstrap.cc:654] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.673341 7386 leader_election.cc:290] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:09.673641 7379 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.673777 7379 ts_tablet_manager.cc:1403] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20251026 08:54:09.674525 7379 raft_consensus.cc:359] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.674746 7379 raft_consensus.cc:385] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.674803 7379 raft_consensus.cc:740] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.675062 7379 consensus_queue.cc:260] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.675676 7379 ts_tablet_manager.cc:1434] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:09.676013 7379 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.677706 7379 tablet_bootstrap.cc:654] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.679410 7377 ts_tablet_manager.cc:1434] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20251026 08:54:09.683480 7379 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.683634 7379 ts_tablet_manager.cc:1403] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20251026 08:54:09.684530 7379 raft_consensus.cc:359] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.684728 7379 raft_consensus.cc:385] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.684792 7379 raft_consensus.cc:740] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.685117 7379 consensus_queue.cc:260] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.688572 7379 ts_tablet_manager.cc:1434] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20251026 08:54:09.693035 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95d18ab1a92641e3bde0717dcc998271" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:09.693217 7183 raft_consensus.cc:2468] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 0.
I20251026 08:54:09.693393 7378 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.693545 7378 ts_tablet_manager.cc:1403] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.011s
I20251026 08:54:09.693629 7298 leader_election.cc:304] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.693929 7393 raft_consensus.cc:2804] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:09.694010 7393 raft_consensus.cc:493] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:09.694124 7393 raft_consensus.cc:3060] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.694262 7378 raft_consensus.cc:359] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.694412 7378 raft_consensus.cc:385] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.694483 7378 raft_consensus.cc:740] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.694679 7378 consensus_queue.cc:260] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.695156 7378 ts_tablet_manager.cc:1434] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:09.695747 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95d18ab1a92641e3bde0717dcc998271" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a" is_pre_election: true
I20251026 08:54:09.695932 7258 raft_consensus.cc:2468] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 0.
I20251026 08:54:09.696563 7393 raft_consensus.cc:515] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.697089 7393 leader_election.cc:290] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:09.697417 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95d18ab1a92641e3bde0717dcc998271" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:09.697592 7183 raft_consensus.cc:3060] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.699542 7183 raft_consensus.cc:2468] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:09.699831 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95d18ab1a92641e3bde0717dcc998271" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a"
I20251026 08:54:09.699957 7258 raft_consensus.cc:3060] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.701627 7258 raft_consensus.cc:2468] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:09.701922 7298 leader_election.cc:304] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.702222 7393 raft_consensus.cc:2804] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:09.702329 7393 raft_consensus.cc:697] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Becoming Leader. State: Replica: b69381bade8d49e8802175e17fa72440, State: Running, Role: LEADER
I20251026 08:54:09.702536 7393 consensus_queue.cc:237] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.704568 7079 catalog_manager.cc:5649] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 reported cstate change: term changed from 0 to 1, leader changed from <none> to b69381bade8d49e8802175e17fa72440 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } }
I20251026 08:54:09.714201 7384 raft_consensus.cc:493] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:09.714310 7384 raft_consensus.cc:515] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.715011 7384 leader_election.cc:290] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:09.720495 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f780788b372466bbc705743cd502bba" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:09.720679 7183 raft_consensus.cc:2468] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 0.
I20251026 08:54:09.721050 7224 leader_election.cc:304] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a; no voters:
I20251026 08:54:09.721289 7384 raft_consensus.cc:2804] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:09.721364 7384 raft_consensus.cc:493] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:09.721451 7384 raft_consensus.cc:3060] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.721704 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f780788b372466bbc705743cd502bba" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440" is_pre_election: true
I20251026 08:54:09.721947 7332 raft_consensus.cc:2468] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 0.
I20251026 08:54:09.723587 7384 raft_consensus.cc:515] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.723960 7384 leader_election.cc:290] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:09.724341 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f780788b372466bbc705743cd502bba" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:09.724354 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f780788b372466bbc705743cd502bba" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440"
I20251026 08:54:09.724470 7183 raft_consensus.cc:3060] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.724515 7332 raft_consensus.cc:3060] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.726764 7183 raft_consensus.cc:2468] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:09.726910 7332 raft_consensus.cc:2468] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:09.727095 7224 leader_election.cc:304] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a; no voters:
I20251026 08:54:09.727335 7384 raft_consensus.cc:2804] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:09.727702 7384 raft_consensus.cc:697] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Becoming Leader. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Running, Role: LEADER
I20251026 08:54:09.727941 7384 consensus_queue.cc:237] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.729720 7079 catalog_manager.cc:5649] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a reported cstate change: term changed from 0 to 1, leader changed from <none> to 618784080c4148edb8e4a00574b8d60a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:09.746618 7384 raft_consensus.cc:493] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:09.746734 7384 raft_consensus.cc:515] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.747162 7384 leader_election.cc:290] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:09.747486 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7b072f91e494d8b995f146c2a9fdc53" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:09.747586 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7b072f91e494d8b995f146c2a9fdc53" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440" is_pre_election: true
I20251026 08:54:09.747669 7183 raft_consensus.cc:2468] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 0.
I20251026 08:54:09.747711 7332 raft_consensus.cc:2468] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 0.
I20251026 08:54:09.748004 7224 leader_election.cc:304] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.748229 7384 raft_consensus.cc:2804] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:09.748301 7384 raft_consensus.cc:493] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:09.748353 7384 raft_consensus.cc:3060] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.750025 7384 raft_consensus.cc:515] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.750527 7384 leader_election.cc:290] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:09.750874 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7b072f91e494d8b995f146c2a9fdc53" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:09.750952 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a7b072f91e494d8b995f146c2a9fdc53" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440"
I20251026 08:54:09.751037 7183 raft_consensus.cc:3060] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.751062 7332 raft_consensus.cc:3060] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.753646 7183 raft_consensus.cc:2468] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:09.753648 7332 raft_consensus.cc:2468] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:09.754060 7224 leader_election.cc:304] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.754307 7384 raft_consensus.cc:2804] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:09.754417 7384 raft_consensus.cc:697] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Becoming Leader. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Running, Role: LEADER
I20251026 08:54:09.754602 7384 consensus_queue.cc:237] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.756585 7079 catalog_manager.cc:5649] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a reported cstate change: term changed from 0 to 1, leader changed from <none> to 618784080c4148edb8e4a00574b8d60a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:09.760424 7382 raft_consensus.cc:493] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:09.760528 7382 raft_consensus.cc:515] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.761119 7382 leader_election.cc:290] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:09.766602 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440" is_pre_election: true
I20251026 08:54:09.766600 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a" is_pre_election: true
I20251026 08:54:09.766762 7332 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 0.
I20251026 08:54:09.766806 7258 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 0.
I20251026 08:54:09.767059 7148 leader_election.cc:304] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.767258 7382 raft_consensus.cc:2804] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:09.767325 7382 raft_consensus.cc:493] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:09.767390 7382 raft_consensus.cc:3060] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.768865 7382 raft_consensus.cc:515] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.769354 7382 leader_election.cc:290] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 election: Requested vote from peers b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:09.769644 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440"
I20251026 08:54:09.769802 7332 raft_consensus.cc:3060] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.769794 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a"
I20251026 08:54:09.769984 7258 raft_consensus.cc:3060] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.771497 7332 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 1.
I20251026 08:54:09.771667 7258 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 1.
I20251026 08:54:09.771814 7148 leader_election.cc:304] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.772007 7382 raft_consensus.cc:2804] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:09.772357 7382 raft_consensus.cc:697] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 LEADER]: Becoming Leader. State: Replica: 16a1439266f547fcb516c394c7025029, State: Running, Role: LEADER
I20251026 08:54:09.772666 7382 consensus_queue.cc:237] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:09.774677 7079 catalog_manager.cc:5649] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16a1439266f547fcb516c394c7025029 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "16a1439266f547fcb516c394c7025029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:09.818717 413 test_util.cc:276] Using random seed: -317486013
I20251026 08:54:09.828706 7079 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39236:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251026 08:54:09.829243 7079 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251026 08:54:09.836313 7171 tablet_service.cc:1505] Processing CreateTablet for tablet 7ddf6b8541b74a3dbb78047caa21b46c (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:54:09.836535 7247 tablet_service.cc:1505] Processing CreateTablet for tablet 7ddf6b8541b74a3dbb78047caa21b46c (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:54:09.836695 7171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddf6b8541b74a3dbb78047caa21b46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.836875 7247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddf6b8541b74a3dbb78047caa21b46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.836858 7322 tablet_service.cc:1505] Processing CreateTablet for tablet 7ddf6b8541b74a3dbb78047caa21b46c (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION VALUES < 536870911
I20251026 08:54:09.837163 7322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ddf6b8541b74a3dbb78047caa21b46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.838474 7170 tablet_service.cc:1505] Processing CreateTablet for tablet 17955cbe1e554a84abecdccab9a62085 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:54:09.838784 7170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17955cbe1e554a84abecdccab9a62085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.840008 7172 tablet_service.cc:1505] Processing CreateTablet for tablet bdfd68ffcf4948b1b230a6959a6dcc85 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:54:09.840359 7172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfd68ffcf4948b1b230a6959a6dcc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.842939 7320 tablet_service.cc:1505] Processing CreateTablet for tablet 17955cbe1e554a84abecdccab9a62085 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:54:09.843088 7379 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.843238 7320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17955cbe1e554a84abecdccab9a62085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.844034 7377 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.844447 7379 tablet_bootstrap.cc:654] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.845503 7377 tablet_bootstrap.cc:654] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.846362 7173 tablet_service.cc:1505] Processing CreateTablet for tablet 82ee312bd84648ada28706a0d6d759f5 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:54:09.846662 7173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ee312bd84648ada28706a0d6d759f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.847052 7319 tablet_service.cc:1505] Processing CreateTablet for tablet 82ee312bd84648ada28706a0d6d759f5 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:54:09.847355 7319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ee312bd84648ada28706a0d6d759f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.850407 7321 tablet_service.cc:1505] Processing CreateTablet for tablet bdfd68ffcf4948b1b230a6959a6dcc85 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:54:09.850745 7321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfd68ffcf4948b1b230a6959a6dcc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.854781 7248 tablet_service.cc:1505] Processing CreateTablet for tablet 17955cbe1e554a84abecdccab9a62085 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20251026 08:54:09.855166 7248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17955cbe1e554a84abecdccab9a62085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.855451 7246 tablet_service.cc:1505] Processing CreateTablet for tablet bdfd68ffcf4948b1b230a6959a6dcc85 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20251026 08:54:09.855755 7246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdfd68ffcf4948b1b230a6959a6dcc85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.860467 7245 tablet_service.cc:1505] Processing CreateTablet for tablet 82ee312bd84648ada28706a0d6d759f5 (DEFAULT_TABLE table=dugtrio [id=6ae977be4f90450daff128564e7a9154]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20251026 08:54:09.860826 7245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ee312bd84648ada28706a0d6d759f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:09.863310 7378 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.864956 7378 tablet_bootstrap.cc:654] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.855168 7379 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.866664 7379 ts_tablet_manager.cc:1403] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.003s
I20251026 08:54:09.867470 7378 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.867691 7378 ts_tablet_manager.cc:1403] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251026 08:54:09.868547 7378 raft_consensus.cc:359] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.868667 7379 raft_consensus.cc:359] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.868742 7378 raft_consensus.cc:385] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.868816 7379 raft_consensus.cc:385] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.868824 7378 raft_consensus.cc:740] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.868872 7379 raft_consensus.cc:740] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.869016 7378 consensus_queue.cc:260] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.869472 7378 ts_tablet_manager.cc:1434] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:54:09.869028 7379 consensus_queue.cc:260] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.869809 7379 ts_tablet_manager.cc:1434] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251026 08:54:09.870098 7379 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.870491 7377 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.870651 7377 ts_tablet_manager.cc:1403] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.011s
I20251026 08:54:09.871465 7379 tablet_bootstrap.cc:654] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.871382 7377 raft_consensus.cc:359] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.871634 7377 raft_consensus.cc:385] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.871727 7377 raft_consensus.cc:740] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.871938 7377 consensus_queue.cc:260] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.872537 7377 ts_tablet_manager.cc:1434] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251026 08:54:09.872766 7377 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.874157 7377 tablet_bootstrap.cc:654] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.876014 7377 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.876223 7377 ts_tablet_manager.cc:1403] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251026 08:54:09.876485 7378 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.877257 7377 raft_consensus.cc:359] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.877381 7377 raft_consensus.cc:385] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.877440 7377 raft_consensus.cc:740] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.877656 7377 consensus_queue.cc:260] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.877971 7378 tablet_bootstrap.cc:654] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.878121 7377 ts_tablet_manager.cc:1434] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:09.878381 7377 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.879271 7379 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.879370 7378 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.879379 7379 ts_tablet_manager.cc:1403] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20251026 08:54:09.879472 7378 ts_tablet_manager.cc:1403] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:09.879686 7377 tablet_bootstrap.cc:654] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.880110 7378 raft_consensus.cc:359] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.880246 7378 raft_consensus.cc:385] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.880290 7378 raft_consensus.cc:740] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.880506 7378 consensus_queue.cc:260] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.880980 7378 ts_tablet_manager.cc:1434] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:09.881083 7377 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.881160 7377 ts_tablet_manager.cc:1403] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:09.881212 7378 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.881654 7377 raft_consensus.cc:359] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.881788 7377 raft_consensus.cc:385] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.881917 7377 raft_consensus.cc:740] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.882215 7377 consensus_queue.cc:260] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.882385 7378 tablet_bootstrap.cc:654] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.882683 7377 ts_tablet_manager.cc:1434] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20251026 08:54:09.882938 7377 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Bootstrap starting.
I20251026 08:54:09.883778 7379 raft_consensus.cc:359] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.884774 7379 raft_consensus.cc:385] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.884877 7379 raft_consensus.cc:740] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.885178 7379 consensus_queue.cc:260] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.885663 7379 ts_tablet_manager.cc:1434] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:54:09.885941 7379 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.886934 7379 tablet_bootstrap.cc:654] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.884338 7377 tablet_bootstrap.cc:654] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.888334 7378 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.888500 7378 ts_tablet_manager.cc:1403] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20251026 08:54:09.889150 7379 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.889228 7377 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: No bootstrap required, opened a new log
I20251026 08:54:09.889215 7378 raft_consensus.cc:359] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.889281 7379 ts_tablet_manager.cc:1403] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:09.889456 7378 raft_consensus.cc:385] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.889454 7377 ts_tablet_manager.cc:1403] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20251026 08:54:09.889573 7378 raft_consensus.cc:740] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.889813 7378 consensus_queue.cc:260] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.890197 7377 raft_consensus.cc:359] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.890282 7379 raft_consensus.cc:359] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.890443 7377 raft_consensus.cc:385] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.890475 7378 ts_tablet_manager.cc:1434] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:09.890573 7377 raft_consensus.cc:740] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69381bade8d49e8802175e17fa72440, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.890607 7379 raft_consensus.cc:385] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.890754 7379 raft_consensus.cc:740] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.890817 7377 consensus_queue.cc:260] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.891083 7378 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Bootstrap starting.
I20251026 08:54:09.891040 7379 consensus_queue.cc:260] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.892095 7377 ts_tablet_manager.cc:1434] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:09.893260 7378 tablet_bootstrap.cc:654] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.894773 7379 ts_tablet_manager.cc:1434] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251026 08:54:09.895077 7378 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: No bootstrap required, opened a new log
I20251026 08:54:09.895242 7378 ts_tablet_manager.cc:1403] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20251026 08:54:09.895541 7379 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:09.896062 7378 raft_consensus.cc:359] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.896296 7378 raft_consensus.cc:385] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.896410 7378 raft_consensus.cc:740] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.896721 7379 tablet_bootstrap.cc:654] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Neither blocks nor log segments found. Creating new log.
I20251026 08:54:09.896629 7378 consensus_queue.cc:260] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.897158 7378 ts_tablet_manager.cc:1434] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251026 08:54:09.898156 7379 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: No bootstrap required, opened a new log
I20251026 08:54:09.898293 7379 ts_tablet_manager.cc:1403] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:09.898777 7379 raft_consensus.cc:359] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.898918 7379 raft_consensus.cc:385] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251026 08:54:09.898991 7379 raft_consensus.cc:740] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:09.899183 7379 consensus_queue.cc:260] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.899570 7379 ts_tablet_manager.cc:1434] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:09.903723 7382 raft_consensus.cc:493] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:09.903854 7382 raft_consensus.cc:515] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.904361 7382 leader_election.cc:290] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:09.904664 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdfd68ffcf4948b1b230a6959a6dcc85" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a" is_pre_election: true
I20251026 08:54:09.904740 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdfd68ffcf4948b1b230a6959a6dcc85" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440" is_pre_election: true
I20251026 08:54:09.904827 7258 raft_consensus.cc:2468] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 0.
I20251026 08:54:09.904963 7332 raft_consensus.cc:2468] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 0.
I20251026 08:54:09.905220 7147 leader_election.cc:304] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a; no voters:
I20251026 08:54:09.905453 7382 raft_consensus.cc:2804] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:09.905534 7382 raft_consensus.cc:493] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:09.905613 7382 raft_consensus.cc:3060] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.907624 7382 raft_consensus.cc:515] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.908061 7382 leader_election.cc:290] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 election: Requested vote from peers 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:09.908370 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdfd68ffcf4948b1b230a6959a6dcc85" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a"
I20251026 08:54:09.908443 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bdfd68ffcf4948b1b230a6959a6dcc85" candidate_uuid: "16a1439266f547fcb516c394c7025029" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440"
I20251026 08:54:09.908537 7258 raft_consensus.cc:3060] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.908581 7332 raft_consensus.cc:3060] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:09.910949 7332 raft_consensus.cc:2468] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 1.
I20251026 08:54:09.910950 7258 raft_consensus.cc:2468] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16a1439266f547fcb516c394c7025029 in term 1.
I20251026 08:54:09.911334 7148 leader_election.cc:304] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:09.911564 7382 raft_consensus.cc:2804] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:09.911712 7382 raft_consensus.cc:697] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 LEADER]: Becoming Leader. State: Replica: 16a1439266f547fcb516c394c7025029, State: Running, Role: LEADER
I20251026 08:54:09.911896 7382 consensus_queue.cc:237] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:09.914031 7078 catalog_manager.cc:5649] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16a1439266f547fcb516c394c7025029 (127.0.103.65). New cstate: current_term: 1 leader_uuid: "16a1439266f547fcb516c394c7025029" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:10.012861 7393 raft_consensus.cc:493] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:10.013051 7393 raft_consensus.cc:515] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.013579 7393 leader_election.cc:290] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:10.013967 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17955cbe1e554a84abecdccab9a62085" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a" is_pre_election: true
I20251026 08:54:10.013947 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17955cbe1e554a84abecdccab9a62085" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:10.014232 7258 raft_consensus.cc:2468] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 0.
I20251026 08:54:10.014269 7183 raft_consensus.cc:2468] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 0.
I20251026 08:54:10.014600 7296 leader_election.cc:304] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 618784080c4148edb8e4a00574b8d60a, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:10.014828 7393 raft_consensus.cc:2804] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:10.014945 7393 raft_consensus.cc:493] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:10.015007 7393 raft_consensus.cc:3060] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.016778 7393 raft_consensus.cc:515] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.017228 7393 leader_election.cc:290] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:10.017563 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17955cbe1e554a84abecdccab9a62085" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:10.017597 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "17955cbe1e554a84abecdccab9a62085" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a"
I20251026 08:54:10.017765 7258 raft_consensus.cc:3060] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.017787 7183 raft_consensus.cc:3060] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.020258 7183 raft_consensus.cc:2468] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:10.020258 7258 raft_consensus.cc:2468] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:10.020618 7296 leader_election.cc:304] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:10.020893 7393 raft_consensus.cc:2804] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:10.021041 7393 raft_consensus.cc:697] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Becoming Leader. State: Replica: b69381bade8d49e8802175e17fa72440, State: Running, Role: LEADER
I20251026 08:54:10.021301 7393 consensus_queue.cc:237] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.023505 7078 catalog_manager.cc:5649] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 reported cstate change: term changed from 0 to 1, leader changed from <none> to b69381bade8d49e8802175e17fa72440 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } }
I20251026 08:54:10.122315 7393 raft_consensus.cc:493] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:10.122503 7393 raft_consensus.cc:515] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.122979 7393 leader_election.cc:290] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:10.123373 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ddf6b8541b74a3dbb78047caa21b46c" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:10.123396 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ddf6b8541b74a3dbb78047caa21b46c" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a" is_pre_election: true
I20251026 08:54:10.123545 7183 raft_consensus.cc:2468] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 0.
I20251026 08:54:10.123576 7258 raft_consensus.cc:2468] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 0.
I20251026 08:54:10.124068 7298 leader_election.cc:304] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:10.124332 7393 raft_consensus.cc:2804] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:10.124419 7393 raft_consensus.cc:493] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:10.124544 7393 raft_consensus.cc:3060] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.126813 7393 raft_consensus.cc:515] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.127355 7393 leader_election.cc:290] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:10.127662 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ddf6b8541b74a3dbb78047caa21b46c" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:10.127699 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7ddf6b8541b74a3dbb78047caa21b46c" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a"
I20251026 08:54:10.127856 7258 raft_consensus.cc:3060] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.127859 7183 raft_consensus.cc:3060] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.130306 7258 raft_consensus.cc:2468] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:10.130347 7183 raft_consensus.cc:2468] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:10.130671 7296 leader_election.cc:304] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:10.130913 7393 raft_consensus.cc:2804] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:10.131072 7393 raft_consensus.cc:697] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Becoming Leader. State: Replica: b69381bade8d49e8802175e17fa72440, State: Running, Role: LEADER
I20251026 08:54:10.131331 7393 consensus_queue.cc:237] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.133386 7078 catalog_manager.cc:5649] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 reported cstate change: term changed from 0 to 1, leader changed from <none> to b69381bade8d49e8802175e17fa72440 (127.0.103.67). New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } }
I20251026 08:54:10.138665 7384 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.143769 7431 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.154932 7431 consensus_queue.cc:1048] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.158272 7384 consensus_queue.cc:1048] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.164847 7393 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.168337 7437 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.236902 7384 raft_consensus.cc:493] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251026 08:54:10.237087 7384 raft_consensus.cc:515] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.237676 7384 leader_election.cc:290] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:10.238111 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ee312bd84648ada28706a0d6d759f5" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:10.238140 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ee312bd84648ada28706a0d6d759f5" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440" is_pre_election: true
I20251026 08:54:10.238305 7183 raft_consensus.cc:2468] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 0.
I20251026 08:54:10.238329 7332 raft_consensus.cc:2468] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 0.
I20251026 08:54:10.238716 7224 leader_election.cc:304] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a, b69381bade8d49e8802175e17fa72440; no voters:
I20251026 08:54:10.238968 7384 raft_consensus.cc:2804] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251026 08:54:10.239058 7384 raft_consensus.cc:493] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251026 08:54:10.239120 7384 raft_consensus.cc:3060] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.241036 7384 raft_consensus.cc:515] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.241499 7384 leader_election.cc:290] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:10.241874 7183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ee312bd84648ada28706a0d6d759f5" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:10.241982 7332 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82ee312bd84648ada28706a0d6d759f5" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440"
I20251026 08:54:10.242086 7183 raft_consensus.cc:3060] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.242154 7332 raft_consensus.cc:3060] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 0 FOLLOWER]: Advancing to term 1
I20251026 08:54:10.243829 7183 raft_consensus.cc:2468] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:10.244071 7332 raft_consensus.cc:2468] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:10.244186 7224 leader_election.cc:304] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a; no voters:
I20251026 08:54:10.244467 7384 raft_consensus.cc:2804] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader election won for term 1
I20251026 08:54:10.244593 7384 raft_consensus.cc:697] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Becoming Leader. State: Replica: 618784080c4148edb8e4a00574b8d60a, State: Running, Role: LEADER
I20251026 08:54:10.244637 7382 consensus_queue.cc:1048] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.244841 7384 consensus_queue.cc:237] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:10.247642 7081 catalog_manager.cc:5649] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a reported cstate change: term changed from 0 to 1, leader changed from <none> to 618784080c4148edb8e4a00574b8d60a (127.0.103.66). New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:10.253682 7441 consensus_queue.cc:1048] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.296159 7441 consensus_queue.cc:1048] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.299669 7441 consensus_queue.cc:1048] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251026 08:54:10.308337 413 tablet_server.cc:178] TabletServer@127.0.103.65:0 shutting down...
I20251026 08:54:10.323443 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:10.323796 413 tablet_replica.cc:333] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.323992 413 raft_consensus.cc:2243] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:10.324131 413 raft_consensus.cc:2272] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.324702 413 tablet_replica.cc:333] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.324810 413 raft_consensus.cc:2243] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:10.324942 413 raft_consensus.cc:2272] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.325476 413 tablet_replica.cc:333] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.325578 413 raft_consensus.cc:2243] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:10.325693 413 raft_consensus.cc:2272] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.326237 413 tablet_replica.cc:333] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.326366 413 raft_consensus.cc:2243] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:10.326498 413 raft_consensus.cc:2272] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.327206 413 tablet_replica.cc:333] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.327355 413 raft_consensus.cc:2243] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:10.327482 413 raft_consensus.cc:2272] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.328110 413 tablet_replica.cc:333] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.328245 413 raft_consensus.cc:2243] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:10.328359 413 raft_consensus.cc:2272] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.328758 413 tablet_replica.cc:333] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.328871 413 raft_consensus.cc:2243] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:10.329102 413 raft_consensus.cc:2272] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.329550 413 tablet_replica.cc:333] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: stopping tablet replica
I20251026 08:54:10.329655 413 raft_consensus.cc:2243] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:10.329864 413 raft_consensus.cc:2272] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:10.344352 413 tablet_server.cc:195] TabletServer@127.0.103.65:0 shutdown complete.
W20251026 08:54:10.458616 7298 consensus_peers.cc:597] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:10.512365 7386 consensus_queue.cc:1048] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:54:10.576448 7224 consensus_peers.cc:597] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:10.592200 7386 consensus_queue.cc:1048] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:54:10.596356 7298 consensus_peers.cc:597] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251026 08:54:10.599781 7298 consensus_peers.cc:597] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20251026 08:54:10.647887 7224 consensus_peers.cc:597] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:10.651086 7384 consensus_queue.cc:1048] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251026 08:54:10.657814 7224 consensus_peers.cc:597] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:11.210258 7393 consensus_queue.cc:579] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Leader has been unable to successfully communicate with peer 16a1439266f547fcb516c394c7025029 for more than 1 seconds (1.042s)
W20251026 08:54:11.212772 7078 catalog_manager.cc:5327] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 95d18ab1a92641e3bde0717dcc998271 with cas_config_opid_index -1: no extra replica candidate found for tablet 95d18ab1a92641e3bde0717dcc998271 (table test-workload [id=37afc5f2418743bb9dbae227c4793b5f]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20251026 08:54:11.243011 7401 consensus_queue.cc:579] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Leader has been unable to successfully communicate with peer 16a1439266f547fcb516c394c7025029 for more than 1 seconds (1.083s)
I20251026 08:54:11.355078 7078 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20251026 08:54:11.355258 7078 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39232:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20251026 08:54:11.359653 7078 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Sending DeleteTablet for 3 replicas of tablet 7ddf6b8541b74a3dbb78047caa21b46c
I20251026 08:54:11.360129 7078 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Sending DeleteTablet for 3 replicas of tablet 17955cbe1e554a84abecdccab9a62085
I20251026 08:54:11.360417 7246 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ddf6b8541b74a3dbb78047caa21b46c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:60806
I20251026 08:54:11.360503 7078 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Sending DeleteTablet for 3 replicas of tablet bdfd68ffcf4948b1b230a6959a6dcc85
I20251026 08:54:11.360913 7078 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: Sending DeleteTablet for 3 replicas of tablet 82ee312bd84648ada28706a0d6d759f5
I20251026 08:54:11.360960 7321 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ddf6b8541b74a3dbb78047caa21b46c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:37438
I20251026 08:54:11.361963 7245 tablet_service.cc:1552] Processing DeleteTablet for tablet 17955cbe1e554a84abecdccab9a62085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:60806
I20251026 08:54:11.361979 7322 tablet_service.cc:1552] Processing DeleteTablet for tablet 82ee312bd84648ada28706a0d6d759f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:37438
I20251026 08:54:11.362358 7320 tablet_service.cc:1552] Processing DeleteTablet for tablet 17955cbe1e554a84abecdccab9a62085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:37438
I20251026 08:54:11.362380 7319 tablet_service.cc:1552] Processing DeleteTablet for tablet bdfd68ffcf4948b1b230a6959a6dcc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:37438
I20251026 08:54:11.362751 7245 tablet_service.cc:1552] Processing DeleteTablet for tablet bdfd68ffcf4948b1b230a6959a6dcc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:60806
W20251026 08:54:11.362959 7129 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 16a1439266f547fcb516c394c7025029 not available for placement
I20251026 08:54:11.363022 7245 tablet_service.cc:1552] Processing DeleteTablet for tablet 82ee312bd84648ada28706a0d6d759f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:60806
I20251026 08:54:11.363131 7462 tablet_replica.cc:333] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:11.363184 7461 tablet_replica.cc:333] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:11.363302 7462 raft_consensus.cc:2243] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:11.363306 7461 raft_consensus.cc:2243] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.363396 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251026 08:54:11.363477 7461 raft_consensus.cc:2272] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.363648 7462 raft_consensus.cc:2272] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.364284 7461 ts_tablet_manager.cc:1916] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.364307 7462 ts_tablet_manager.cc:1916] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251026 08:54:11.366281 7068 catalog_manager.cc:4712] TS 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 82ee312bd84648ada28706a0d6d759f5: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111)
W20251026 08:54:11.366715 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:11.367836 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:11.370649 7461 ts_tablet_manager.cc:1929] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.370649 7462 ts_tablet_manager.cc:1929] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.370872 7461 log.cc:1199] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/wals/7ddf6b8541b74a3dbb78047caa21b46c
I20251026 08:54:11.370930 7462 log.cc:1199] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root/wals/7ddf6b8541b74a3dbb78047caa21b46c
I20251026 08:54:11.371346 7462 ts_tablet_manager.cc:1950] T 7ddf6b8541b74a3dbb78047caa21b46c P b69381bade8d49e8802175e17fa72440: Deleting consensus metadata
W20251026 08:54:11.371978 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:11.372171 413 server_base.cc:1047] running on GCE node
I20251026 08:54:11.372336 7462 tablet_replica.cc:333] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:11.372360 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251026 08:54:11.372383 7067 catalog_manager.cc:4985] TS b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829): tablet 7ddf6b8541b74a3dbb78047caa21b46c (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
W20251026 08:54:11.372467 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:11.372499 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468851372497 us; error 0 us; skew 500 ppm
I20251026 08:54:11.372517 7462 raft_consensus.cc:2243] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.372650 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:11.372668 7462 raft_consensus.cc:2272] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.371362 7461 ts_tablet_manager.cc:1950] T 7ddf6b8541b74a3dbb78047caa21b46c P 618784080c4148edb8e4a00574b8d60a: Deleting consensus metadata
I20251026 08:54:11.373277 7462 ts_tablet_manager.cc:1916] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.373688 7461 tablet_replica.cc:333] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:11.373776 7066 catalog_manager.cc:4985] TS 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511): tablet 7ddf6b8541b74a3dbb78047caa21b46c (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.373893 7461 raft_consensus.cc:2243] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.374032 7461 raft_consensus.cc:2272] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.374842 7461 ts_tablet_manager.cc:1916] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.375046 413 webserver.cc:492] Webserver started at http://127.0.103.68:33053/ using document root <none> and password file <none>
I20251026 08:54:11.375311 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:11.375373 413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251026 08:54:11.375475 413 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251026 08:54:11.376101 413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root/instance:
uuid: "b306efaa85e34d228d4ecee71b794136"
format_stamp: "Formatted at 2025-10-26 08:54:11 on dist-test-slave-rd83"
I20251026 08:54:11.378710 7461 ts_tablet_manager.cc:1929] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.378852 7461 log.cc:1199] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/wals/17955cbe1e554a84abecdccab9a62085
I20251026 08:54:11.378927 7462 ts_tablet_manager.cc:1929] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.378974 413 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20251026 08:54:11.379055 7462 log.cc:1199] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root/wals/82ee312bd84648ada28706a0d6d759f5
I20251026 08:54:11.379253 7461 ts_tablet_manager.cc:1950] T 17955cbe1e554a84abecdccab9a62085 P 618784080c4148edb8e4a00574b8d60a: Deleting consensus metadata
I20251026 08:54:11.379427 7462 ts_tablet_manager.cc:1950] T 82ee312bd84648ada28706a0d6d759f5 P b69381bade8d49e8802175e17fa72440: Deleting consensus metadata
I20251026 08:54:11.380003 7461 tablet_replica.cc:333] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:11.380105 7066 catalog_manager.cc:4985] TS 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511): tablet 17955cbe1e554a84abecdccab9a62085 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.380210 7461 raft_consensus.cc:2243] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.380267 7067 catalog_manager.cc:4985] TS b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829): tablet 82ee312bd84648ada28706a0d6d759f5 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.380335 7462 tablet_replica.cc:333] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:11.380396 7461 raft_consensus.cc:2272] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.380470 7462 raft_consensus.cc:2243] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:11.380725 7462 raft_consensus.cc:2272] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.381086 7461 ts_tablet_manager.cc:1916] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.381501 7462 ts_tablet_manager.cc:1916] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.382344 7472 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:11.382699 413 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251026 08:54:11.382802 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root
uuid: "b306efaa85e34d228d4ecee71b794136"
format_stamp: "Formatted at 2025-10-26 08:54:11 on dist-test-slave-rd83"
I20251026 08:54:11.382934 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:11.386360 7462 ts_tablet_manager.cc:1929] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.386462 7462 log.cc:1199] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root/wals/17955cbe1e554a84abecdccab9a62085
I20251026 08:54:11.386749 7462 ts_tablet_manager.cc:1950] T 17955cbe1e554a84abecdccab9a62085 P b69381bade8d49e8802175e17fa72440: Deleting consensus metadata
I20251026 08:54:11.386807 7461 ts_tablet_manager.cc:1929] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.386909 7461 log.cc:1199] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/wals/bdfd68ffcf4948b1b230a6959a6dcc85
I20251026 08:54:11.387204 7461 ts_tablet_manager.cc:1950] T bdfd68ffcf4948b1b230a6959a6dcc85 P 618784080c4148edb8e4a00574b8d60a: Deleting consensus metadata
I20251026 08:54:11.387413 7067 catalog_manager.cc:4985] TS b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829): tablet 17955cbe1e554a84abecdccab9a62085 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.387483 7462 tablet_replica.cc:333] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:11.387606 7462 raft_consensus.cc:2243] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.387719 7462 raft_consensus.cc:2272] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.387912 7066 catalog_manager.cc:4985] TS 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511): tablet bdfd68ffcf4948b1b230a6959a6dcc85 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.388135 7462 ts_tablet_manager.cc:1916] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.388146 7461 tablet_replica.cc:333] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:11.388298 7461 raft_consensus.cc:2243] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:11.388536 7461 raft_consensus.cc:2272] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.389263 7461 ts_tablet_manager.cc:1916] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.391840 7462 ts_tablet_manager.cc:1929] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.391925 7462 log.cc:1199] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-2-root/wals/bdfd68ffcf4948b1b230a6959a6dcc85
I20251026 08:54:11.392201 7462 ts_tablet_manager.cc:1950] T bdfd68ffcf4948b1b230a6959a6dcc85 P b69381bade8d49e8802175e17fa72440: Deleting consensus metadata
I20251026 08:54:11.392866 7067 catalog_manager.cc:4985] TS b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829): tablet bdfd68ffcf4948b1b230a6959a6dcc85 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.394608 7461 ts_tablet_manager.cc:1929] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.394695 7461 log.cc:1199] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/wals/82ee312bd84648ada28706a0d6d759f5
I20251026 08:54:11.394963 7461 ts_tablet_manager.cc:1950] T 82ee312bd84648ada28706a0d6d759f5 P 618784080c4148edb8e4a00574b8d60a: Deleting consensus metadata
I20251026 08:54:11.395790 7066 catalog_manager.cc:4985] TS 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511): tablet 82ee312bd84648ada28706a0d6d759f5 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.403311 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:11.403718 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:11.404114 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:11.404855 413 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251026 08:54:11.404906 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:11.404973 413 ts_tablet_manager.cc:616] Registered 0 tablets
I20251026 08:54:11.405037 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:11.415261 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.68:36793
I20251026 08:54:11.415273 7534 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.68:36793 every 8 connection(s)
I20251026 08:54:11.416684 413 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251026 08:54:11.419570 7538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251026 08:54:11.419677 7539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:11.420995 413 server_base.cc:1047] running on GCE node
W20251026 08:54:11.422189 7541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251026 08:54:11.422775 413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20251026 08:54:11.422857 7535 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36111
I20251026 08:54:11.422982 7535 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:11.423238 7535 heartbeater.cc:507] Master 127.0.103.126:36111 requested a full tablet report, sending...
W20251026 08:54:11.422868 413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20251026 08:54:11.423509 413 hybrid_clock.cc:648] HybridClock initialized: now 1761468851423488 us; error 0 us; skew 500 ppm
I20251026 08:54:11.423660 413 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251026 08:54:11.424124 7078 ts_manager.cc:194] Registered new tserver with Master: b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793)
I20251026 08:54:11.424818 413 webserver.cc:492] Webserver started at http://127.0.103.65:36929/ using document root <none> and password file <none>
I20251026 08:54:11.425071 413 fs_manager.cc:362] Metadata directory not provided
I20251026 08:54:11.425069 7078 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39268
I20251026 08:54:11.425184 413 fs_manager.cc:365] Using existing metadata directory in first data directory
I20251026 08:54:11.427228 413 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20251026 08:54:11.428766 7546 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251026 08:54:11.429126 413 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:11.429205 413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root
uuid: "16a1439266f547fcb516c394c7025029"
format_stamp: "Formatted at 2025-10-26 08:54:08 on dist-test-slave-rd83"
I20251026 08:54:11.429327 413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251026 08:54:11.448576 413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251026 08:54:11.449051 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251026 08:54:11.449467 413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251026 08:54:11.450567 7553 ts_tablet_manager.cc:548] Loading tablet metadata (0/8 complete)
I20251026 08:54:11.463752 413 ts_tablet_manager.cc:585] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20251026 08:54:11.463835 413 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.014s user 0.001s sys 0.000s
I20251026 08:54:11.463925 413 ts_tablet_manager.cc:600] Registering tablets (0/8 complete)
I20251026 08:54:11.465735 7553 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.468883 7553 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.469218 7553 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.469393 7553 ts_tablet_manager.cc:1403] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20251026 08:54:11.469820 7553 raft_consensus.cc:359] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.469931 7553 raft_consensus.cc:740] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.470117 7553 consensus_queue.cc:260] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.470593 7553 ts_tablet_manager.cc:1434] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:54:11.470757 7553 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.473132 7553 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.473361 7553 tablet_bootstrap.cc:492] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.473520 7553 ts_tablet_manager.cc:1403] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:11.473912 7553 raft_consensus.cc:359] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.474014 7553 raft_consensus.cc:740] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.474176 7553 consensus_queue.cc:260] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.474453 7553 ts_tablet_manager.cc:1434] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:11.474606 7553 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.476050 413 ts_tablet_manager.cc:616] Registered 8 tablets
I20251026 08:54:11.476222 413 ts_tablet_manager.cc:595] Time spent register tablets: real 0.012s user 0.010s sys 0.000s
I20251026 08:54:11.477109 7553 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.477414 7553 tablet_bootstrap.cc:492] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.477617 7553 ts_tablet_manager.cc:1403] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20251026 08:54:11.478127 7553 raft_consensus.cc:359] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.478308 7553 raft_consensus.cc:740] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.478475 7553 consensus_queue.cc:260] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.478852 7553 ts_tablet_manager.cc:1434] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:54:11.478757 7257 consensus_queue.cc:237] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:11.479036 7553 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.481899 7331 raft_consensus.cc:1275] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:54:11.482532 7384 consensus_queue.cc:1048] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:11.487918 7384 raft_consensus.cc:2955] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.490736 7331 raft_consensus.cc:2955] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
W20251026 08:54:11.493865 7224 consensus_peers.cc:597] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:11.494346 7553 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.494532 7553 tablet_bootstrap.cc:492] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.494654 7553 ts_tablet_manager.cc:1403] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.004s
I20251026 08:54:11.495096 7553 raft_consensus.cc:359] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.495327 7553 raft_consensus.cc:740] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.495604 7553 consensus_queue.cc:260] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.496057 7553 ts_tablet_manager.cc:1434] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251026 08:54:11.496358 7553 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.496345 7078 catalog_manager.cc:5649] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 reported cstate change: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.498301 7066 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a7b072f91e494d8b995f146c2a9fdc53 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20251026 08:54:11.507445 7553 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20251026 08:54:11.507591 7224 consensus_peers.cc:597] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a -> Peer b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): Couldn't send request to peer b306efaa85e34d228d4ecee71b794136. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7b072f91e494d8b995f146c2a9fdc53. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:11.507802 7553 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.507970 7553 ts_tablet_manager.cc:1403] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20251026 08:54:11.508389 7553 raft_consensus.cc:359] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.508499 7553 raft_consensus.cc:740] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.508754 7553 consensus_queue.cc:260] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.509268 7553 ts_tablet_manager.cc:1434] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251026 08:54:11.509485 7553 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.512923 7553 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.513240 7553 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.513415 7553 ts_tablet_manager.cc:1403] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251026 08:54:11.513979 7553 raft_consensus.cc:359] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.514168 7553 raft_consensus.cc:740] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.514367 7553 consensus_queue.cc:260] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.514725 7553 ts_tablet_manager.cc:1434] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251026 08:54:11.514895 7553 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.518119 7553 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.518497 7553 tablet_bootstrap.cc:492] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.518674 7553 ts_tablet_manager.cc:1403] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20251026 08:54:11.519058 7553 raft_consensus.cc:359] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.519157 7553 raft_consensus.cc:740] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.519327 7553 consensus_queue.cc:260] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:11.519621 7553 ts_tablet_manager.cc:1434] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:11.519793 7553 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Bootstrap starting.
I20251026 08:54:11.523280 7553 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.523562 7553 tablet_bootstrap.cc:492] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Bootstrap complete.
I20251026 08:54:11.523788 7553 ts_tablet_manager.cc:1403] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20251026 08:54:11.524221 7553 raft_consensus.cc:359] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.524345 7553 raft_consensus.cc:740] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16a1439266f547fcb516c394c7025029, State: Initialized, Role: FOLLOWER
I20251026 08:54:11.524576 7553 consensus_queue.cc:260] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.525019 7553 ts_tablet_manager.cc:1434] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251026 08:54:11.527252 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.103.65:41911
I20251026 08:54:11.527267 7622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.103.65:41911 every 8 connection(s)
I20251026 08:54:11.532999 7623 heartbeater.cc:344] Connected to a master server at 127.0.103.126:36111
I20251026 08:54:11.533092 7623 heartbeater.cc:461] Registering TS with master...
I20251026 08:54:11.533398 7623 heartbeater.cc:507] Master 127.0.103.126:36111 requested a full tablet report, sending...
I20251026 08:54:11.534543 7078 ts_manager.cc:194] Re-registered known tserver with Master: 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911)
I20251026 08:54:11.536511 7586 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ddf6b8541b74a3dbb78047caa21b46c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:53786
I20251026 08:54:11.537047 7627 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:11.537178 7627 raft_consensus.cc:2243] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.537294 7627 raft_consensus.cc:2272] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.537564 7078 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:39282
I20251026 08:54:11.537798 7627 ts_tablet_manager.cc:1916] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.542150 7627 ts_tablet_manager.cc:1929] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 08:54:11.542222 7627 log.cc:1199] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/wals/7ddf6b8541b74a3dbb78047caa21b46c
I20251026 08:54:11.542440 7627 ts_tablet_manager.cc:1950] T 7ddf6b8541b74a3dbb78047caa21b46c P 16a1439266f547fcb516c394c7025029: Deleting consensus metadata
I20251026 08:54:11.543102 7068 catalog_manager.cc:4985] TS 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): tablet 7ddf6b8541b74a3dbb78047caa21b46c (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.556686 7331 consensus_queue.cc:237] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:11.558982 7257 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:54:11.559448 7453 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:11.561511 7384 consensus_queue.cc:579] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Leader has been unable to successfully communicate with peer 16a1439266f547fcb516c394c7025029 for more than 1 seconds (1.418s)
I20251026 08:54:11.570432 7258 consensus_queue.cc:237] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
W20251026 08:54:11.574563 7298 consensus_peers.cc:597] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 -> Peer b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): Couldn't send request to peer b306efaa85e34d228d4ecee71b794136. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95d18ab1a92641e3bde0717dcc998271. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:11.576771 7331 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:54:11.576548 7453 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
W20251026 08:54:11.576877 7224 consensus_peers.cc:597] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a -> Peer b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): Couldn't send request to peer b306efaa85e34d228d4ecee71b794136. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f780788b372466bbc705743cd502bba. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:11.577306 7401 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:11.580331 7067 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 95d18ab1a92641e3bde0717dcc998271 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20251026 08:54:11.581041 7258 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.582335 7078 catalog_manager.cc:5649] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 reported cstate change: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:11.584774 7586 tablet_service.cc:1552] Processing DeleteTablet for tablet 82ee312bd84648ada28706a0d6d759f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:53786
I20251026 08:54:11.585003 7627 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:11.585098 7623 heartbeater.cc:499] Master 127.0.103.126:36111 was elected leader, sending a full tablet report...
I20251026 08:54:11.585194 7627 raft_consensus.cc:2243] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.585213 7595 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:54:11.585322 7627 raft_consensus.cc:2272] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.585877 7594 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251026 08:54:11.586140 7627 ts_tablet_manager.cc:1916] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.590929 7627 ts_tablet_manager.cc:1929] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 08:54:11.591032 7627 log.cc:1199] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/wals/82ee312bd84648ada28706a0d6d759f5
I20251026 08:54:11.590806 7401 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.593084 7331 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.594388 7066 catalog_manager.cc:5162] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8f780788b372466bbc705743cd502bba with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20251026 08:54:11.596233 7453 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:11.596496 7585 tablet_service.cc:1552] Processing DeleteTablet for tablet bdfd68ffcf4948b1b230a6959a6dcc85 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:53786
I20251026 08:54:11.596654 7586 tablet_service.cc:1552] Processing DeleteTablet for tablet 17955cbe1e554a84abecdccab9a62085 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-26 08:54:11 UTC) from {username='slave'} at 127.0.0.1:53786
I20251026 08:54:11.596887 7431 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:11.597220 7079 catalog_manager.cc:5649] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a reported cstate change: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:11.598852 7595 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.600195 7594 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:11.604871 7627 ts_tablet_manager.cc:1950] T 82ee312bd84648ada28706a0d6d759f5 P 16a1439266f547fcb516c394c7025029: Deleting consensus metadata
I20251026 08:54:11.605819 7627 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:11.605857 7068 catalog_manager.cc:4985] TS 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): tablet 82ee312bd84648ada28706a0d6d759f5 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.605998 7627 raft_consensus.cc:2243] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.606135 7627 raft_consensus.cc:2272] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.606622 7627 ts_tablet_manager.cc:1916] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.609747 7627 ts_tablet_manager.cc:1929] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251026 08:54:11.609823 7627 log.cc:1199] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/wals/bdfd68ffcf4948b1b230a6959a6dcc85
I20251026 08:54:11.610100 7627 ts_tablet_manager.cc:1950] T bdfd68ffcf4948b1b230a6959a6dcc85 P 16a1439266f547fcb516c394c7025029: Deleting consensus metadata
I20251026 08:54:11.610693 7627 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:11.610831 7627 raft_consensus.cc:2243] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:11.610786 7068 catalog_manager.cc:4985] TS 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): tablet bdfd68ffcf4948b1b230a6959a6dcc85 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.610946 7627 raft_consensus.cc:2272] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:11.611406 7627 ts_tablet_manager.cc:1916] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251026 08:54:11.614631 7627 ts_tablet_manager.cc:1929] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251026 08:54:11.614707 7627 log.cc:1199] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/wals/17955cbe1e554a84abecdccab9a62085
I20251026 08:54:11.614907 7627 ts_tablet_manager.cc:1950] T 17955cbe1e554a84abecdccab9a62085 P 16a1439266f547fcb516c394c7025029: Deleting consensus metadata
I20251026 08:54:11.615556 7068 catalog_manager.cc:4985] TS 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): tablet 17955cbe1e554a84abecdccab9a62085 (table dugtrio [id=6ae977be4f90450daff128564e7a9154]) successfully deleted
I20251026 08:54:11.753639 7453 raft_consensus.cc:493] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16a1439266f547fcb516c394c7025029)
I20251026 08:54:11.753803 7453 raft_consensus.cc:515] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.754371 7453 leader_election.cc:290] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:11.754844 7594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:11.754868 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a" is_pre_election: true
I20251026 08:54:11.755014 7594 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 1.
I20251026 08:54:11.755435 7298 leader_election.cc:304] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters: 618784080c4148edb8e4a00574b8d60a
I20251026 08:54:11.755633 7453 raft_consensus.cc:2804] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:54:11.755719 7453 raft_consensus.cc:493] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16a1439266f547fcb516c394c7025029)
I20251026 08:54:11.755790 7453 raft_consensus.cc:3060] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:11.756269 7401 raft_consensus.cc:493] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 16a1439266f547fcb516c394c7025029)
I20251026 08:54:11.756371 7401 raft_consensus.cc:515] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.756836 7401 leader_election.cc:290] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:11.757145 7594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029" is_pre_election: true
I20251026 08:54:11.757265 7331 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440" is_pre_election: true
I20251026 08:54:11.757300 7594 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 618784080c4148edb8e4a00574b8d60a in term 1.
I20251026 08:54:11.757623 7224 leader_election.cc:304] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16a1439266f547fcb516c394c7025029, 618784080c4148edb8e4a00574b8d60a; no voters:
I20251026 08:54:11.757824 7401 raft_consensus.cc:2804] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251026 08:54:11.757901 7401 raft_consensus.cc:493] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 16a1439266f547fcb516c394c7025029)
I20251026 08:54:11.757974 7401 raft_consensus.cc:3060] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:11.758196 7453 raft_consensus.cc:515] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.758425 7331 raft_consensus.cc:2393] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 618784080c4148edb8e4a00574b8d60a in current term 2: Already voted for candidate b69381bade8d49e8802175e17fa72440 in this term.
I20251026 08:54:11.758653 7453 leader_election.cc:290] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 2 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:11.759003 7258 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "618784080c4148edb8e4a00574b8d60a"
I20251026 08:54:11.759166 7594 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "b69381bade8d49e8802175e17fa72440" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:11.759301 7594 raft_consensus.cc:3060] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Advancing to term 2
I20251026 08:54:11.760269 7401 raft_consensus.cc:515] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.760721 7401 leader_election.cc:290] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 2 election: Requested vote from peers 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911), b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:11.760962 7258 raft_consensus.cc:2393] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b69381bade8d49e8802175e17fa72440 in current term 2: Already voted for candidate 618784080c4148edb8e4a00574b8d60a in this term.
I20251026 08:54:11.761075 7595 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "16a1439266f547fcb516c394c7025029"
I20251026 08:54:11.761129 7331 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86e6554eb93645fe95124508aa0a4171" candidate_uuid: "618784080c4148edb8e4a00574b8d60a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b69381bade8d49e8802175e17fa72440"
I20251026 08:54:11.761298 7331 raft_consensus.cc:2393] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 618784080c4148edb8e4a00574b8d60a in current term 2: Already voted for candidate b69381bade8d49e8802175e17fa72440 in this term.
I20251026 08:54:11.761600 7223 leader_election.cc:304] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 618784080c4148edb8e4a00574b8d60a; no voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440
I20251026 08:54:11.761826 7401 raft_consensus.cc:2749] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251026 08:54:11.762441 7594 raft_consensus.cc:2468] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b69381bade8d49e8802175e17fa72440 in term 2.
I20251026 08:54:11.762688 7298 leader_election.cc:304] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 16a1439266f547fcb516c394c7025029, b69381bade8d49e8802175e17fa72440; no voters: 618784080c4148edb8e4a00574b8d60a
I20251026 08:54:11.762888 7453 raft_consensus.cc:2804] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 FOLLOWER]: Leader election won for term 2
I20251026 08:54:11.762976 7453 raft_consensus.cc:697] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 LEADER]: Becoming Leader. State: Replica: b69381bade8d49e8802175e17fa72440, State: Running, Role: LEADER
I20251026 08:54:11.763175 7453 consensus_queue.cc:237] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } }
I20251026 08:54:11.765009 7078 catalog_manager.cc:5649] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 reported cstate change: term changed from 1 to 2, leader changed from 16a1439266f547fcb516c394c7025029 (127.0.103.65) to b69381bade8d49e8802175e17fa72440 (127.0.103.67). New cstate: current_term: 2 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:11.968987 7637 ts_tablet_manager.cc:933] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Initiating tablet copy from peer 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:11.969486 7637 tablet_copy_client.cc:323] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:41511
I20251026 08:54:11.974327 7268 tablet_copy_service.cc:140] P 618784080c4148edb8e4a00574b8d60a: Received BeginTabletCopySession request for tablet a7b072f91e494d8b995f146c2a9fdc53 from peer b306efaa85e34d228d4ecee71b794136 ({username='slave'} at 127.0.0.1:60836)
I20251026 08:54:11.974439 7268 tablet_copy_service.cc:161] P 618784080c4148edb8e4a00574b8d60a: Beginning new tablet copy session on tablet a7b072f91e494d8b995f146c2a9fdc53 from peer b306efaa85e34d228d4ecee71b794136 at {username='slave'} at 127.0.0.1:60836: session id = b306efaa85e34d228d4ecee71b794136-a7b072f91e494d8b995f146c2a9fdc53
I20251026 08:54:11.976414 7268 tablet_copy_source_session.cc:215] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:11.977404 7637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7b072f91e494d8b995f146c2a9fdc53. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:11.981032 7637 tablet_copy_client.cc:806] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:11.981246 7637 tablet_copy_client.cc:670] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:11.982635 7637 tablet_copy_client.cc:538] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:11.984522 7637 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Bootstrap starting.
I20251026 08:54:11.988298 7637 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:11.988540 7637 tablet_bootstrap.cc:492] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Bootstrap complete.
I20251026 08:54:11.988711 7637 ts_tablet_manager.cc:1403] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20251026 08:54:11.989171 7637 raft_consensus.cc:359] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:11.989336 7637 raft_consensus.cc:740] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b306efaa85e34d228d4ecee71b794136, State: Initialized, Role: LEARNER
I20251026 08:54:11.989476 7637 consensus_queue.cc:260] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:11.989961 7637 ts_tablet_manager.cc:1434] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:54:11.990015 7535 heartbeater.cc:499] Master 127.0.103.126:36111 was elected leader, sending a full tablet report...
I20251026 08:54:11.990523 7268 tablet_copy_service.cc:342] P 618784080c4148edb8e4a00574b8d60a: Request end of tablet copy session b306efaa85e34d228d4ecee71b794136-a7b072f91e494d8b995f146c2a9fdc53 received from {username='slave'} at 127.0.0.1:60836
I20251026 08:54:11.990614 7268 tablet_copy_service.cc:434] P 618784080c4148edb8e4a00574b8d60a: ending tablet copy session b306efaa85e34d228d4ecee71b794136-a7b072f91e494d8b995f146c2a9fdc53 on tablet a7b072f91e494d8b995f146c2a9fdc53 with peer b306efaa85e34d228d4ecee71b794136
I20251026 08:54:12.008023 7331 consensus_queue.cc:237] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:12.009313 7258 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:54:12.009335 7594 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:54:12.009706 7393 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:12.009907 7453 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:12.011523 7452 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.012005 7258 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.014045 7067 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 95d18ab1a92641e3bde0717dcc998271 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:54:12.014430 7594 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.014950 7078 catalog_manager.cc:5649] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 reported cstate change: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } }
I20251026 08:54:12.022675 7594 raft_consensus.cc:2955] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.023315 7500 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d18ab1a92641e3bde0717dcc998271 with delete_type TABLET_DATA_TOMBSTONED (TS b306efaa85e34d228d4ecee71b794136 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36002
W20251026 08:54:12.024122 7068 catalog_manager.cc:4952] TS b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): delete failed for tablet 95d18ab1a92641e3bde0717dcc998271 because the tablet was not found. No further retry: Not found: Tablet not found: 95d18ab1a92641e3bde0717dcc998271
I20251026 08:54:12.089489 7637 ts_tablet_manager.cc:933] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Initiating tablet copy from peer 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:12.090005 7637 tablet_copy_client.cc:323] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:41511
I20251026 08:54:12.090548 7268 tablet_copy_service.cc:140] P 618784080c4148edb8e4a00574b8d60a: Received BeginTabletCopySession request for tablet 8f780788b372466bbc705743cd502bba from peer b306efaa85e34d228d4ecee71b794136 ({username='slave'} at 127.0.0.1:60836)
I20251026 08:54:12.090648 7268 tablet_copy_service.cc:161] P 618784080c4148edb8e4a00574b8d60a: Beginning new tablet copy session on tablet 8f780788b372466bbc705743cd502bba from peer b306efaa85e34d228d4ecee71b794136 at {username='slave'} at 127.0.0.1:60836: session id = b306efaa85e34d228d4ecee71b794136-8f780788b372466bbc705743cd502bba
I20251026 08:54:12.092331 7268 tablet_copy_source_session.cc:215] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:12.093299 7637 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f780788b372466bbc705743cd502bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:12.096951 7637 tablet_copy_client.cc:806] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:12.097107 7637 tablet_copy_client.cc:670] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:12.098233 7637 tablet_copy_client.cc:538] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:12.100210 7637 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Bootstrap starting.
I20251026 08:54:12.104139 7637 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:12.104399 7637 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Bootstrap complete.
I20251026 08:54:12.104571 7637 ts_tablet_manager.cc:1403] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Time spent bootstrapping tablet: real 0.004s user 0.005s sys 0.000s
I20251026 08:54:12.105120 7637 raft_consensus.cc:359] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.105266 7637 raft_consensus.cc:740] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b306efaa85e34d228d4ecee71b794136, State: Initialized, Role: LEARNER
I20251026 08:54:12.105417 7637 consensus_queue.cc:260] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.105798 7637 ts_tablet_manager.cc:1434] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:12.106473 7268 tablet_copy_service.cc:342] P 618784080c4148edb8e4a00574b8d60a: Request end of tablet copy session b306efaa85e34d228d4ecee71b794136-8f780788b372466bbc705743cd502bba received from {username='slave'} at 127.0.0.1:60836
I20251026 08:54:12.106559 7268 tablet_copy_service.cc:434] P 618784080c4148edb8e4a00574b8d60a: ending tablet copy session b306efaa85e34d228d4ecee71b794136-8f780788b372466bbc705743cd502bba on tablet 8f780788b372466bbc705743cd502bba with peer b306efaa85e34d228d4ecee71b794136
I20251026 08:54:12.188663 7258 consensus_queue.cc:237] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:12.189985 7594 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:54:12.190011 7331 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:54:12.190402 7647 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:12.190531 7648 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:12.192306 7401 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.192902 7331 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.193002 7594 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.195892 7066 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 8f780788b372466bbc705743cd502bba with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:54:12.196676 7081 catalog_manager.cc:5649] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 reported cstate change: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.199309 7500 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f780788b372466bbc705743cd502bba with delete_type TABLET_DATA_TOMBSTONED (TS b306efaa85e34d228d4ecee71b794136 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36002
I20251026 08:54:12.199580 7644 tablet_replica.cc:333] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: stopping tablet replica
I20251026 08:54:12.199728 7644 raft_consensus.cc:2243] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Raft consensus shutting down.
I20251026 08:54:12.199858 7644 raft_consensus.cc:2272] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Raft consensus is shut down!
I20251026 08:54:12.200410 7644 ts_tablet_manager.cc:1916] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:54:12.204131 7644 ts_tablet_manager.cc:1929] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20251026 08:54:12.204222 7644 log.cc:1199] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root/wals/8f780788b372466bbc705743cd502bba
I20251026 08:54:12.204695 7068 catalog_manager.cc:4985] TS b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): tablet 8f780788b372466bbc705743cd502bba (table test-workload [id=37afc5f2418743bb9dbae227c4793b5f]) successfully deleted
I20251026 08:54:12.237353 7258 raft_consensus.cc:1275] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 2 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:12.237869 7453 consensus_queue.cc:1048] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:12.241021 7594 raft_consensus.cc:1275] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 2 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20251026 08:54:12.241611 7452 consensus_queue.cc:1048] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20251026 08:54:12.370231 7258 consensus_queue.cc:237] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.371814 7331 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:54:12.371814 7594 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20251026 08:54:12.371979 7224 consensus_peers.cc:597] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a -> Peer b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): Couldn't send request to peer b306efaa85e34d228d4ecee71b794136. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:12.372359 7648 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:54:12.372556 7401 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:54:12.374253 7648 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.374766 7594 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.374691 7331 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.379160 7081 catalog_manager.cc:5649] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 reported cstate change: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.385417 7331 consensus_queue.cc:237] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
W20251026 08:54:12.387717 7298 consensus_peers.cc:597] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 -> Peer b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): Couldn't send request to peer b306efaa85e34d228d4ecee71b794136. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95d18ab1a92641e3bde0717dcc998271. This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:12.387727 7258 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:54:12.388015 7594 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20251026 08:54:12.388229 7453 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:54:12.388484 7393 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20251026 08:54:12.390101 7452 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.390565 7594 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
I20251026 08:54:12.391126 7258 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
W20251026 08:54:12.393899 7129 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID b306efaa85e34d228d4ecee71b794136 is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.394555 7078 catalog_manager.cc:5649] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a reported cstate change: config changed from index 3 to 4, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) added. New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } } }
W20251026 08:54:12.398610 7129 auto_rebalancer.cc:663] Could not move replica: Illegal state: a7b072f91e494d8b995f146c2a9fdc53: source replica b69381bade8d49e8802175e17fa72440 does not have REPLACE attribute set
W20251026 08:54:12.398670 7129 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: a7b072f91e494d8b995f146c2a9fdc53: source replica b69381bade8d49e8802175e17fa72440 does not have REPLACE attribute set
I20251026 08:54:12.471998 7510 raft_consensus.cc:1217] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20251026 08:54:12.583483 7257 consensus_queue.cc:237] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } }
I20251026 08:54:12.585268 7331 raft_consensus.cc:1275] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:54:12.585284 7594 raft_consensus.cc:1275] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20251026 08:54:12.585717 7401 consensus_queue.cc:1048] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:12.585896 7655 consensus_queue.cc:1048] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251026 08:54:12.587716 7401 raft_consensus.cc:2955] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.588287 7594 raft_consensus.cc:2955] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.590786 7066 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet a7b072f91e494d8b995f146c2a9fdc53 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:54:12.591751 7079 catalog_manager.cc:5649] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a reported cstate change: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:12.592746 7331 raft_consensus.cc:2955] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER b306efaa85e34d228d4ecee71b794136 (127.0.103.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } }
I20251026 08:54:12.594614 7500 tablet_service.cc:1552] Processing DeleteTablet for tablet a7b072f91e494d8b995f146c2a9fdc53 with delete_type TABLET_DATA_TOMBSTONED (TS b306efaa85e34d228d4ecee71b794136 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:36002
I20251026 08:54:12.594934 7644 tablet_replica.cc:333] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: stopping tablet replica
I20251026 08:54:12.595084 7644 raft_consensus.cc:2243] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Raft consensus shutting down.
I20251026 08:54:12.595252 7644 raft_consensus.cc:2272] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Raft consensus is shut down!
I20251026 08:54:12.595834 7644 ts_tablet_manager.cc:1916] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:54:12.600860 7644 ts_tablet_manager.cc:1929] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20251026 08:54:12.600960 7644 log.cc:1199] T a7b072f91e494d8b995f146c2a9fdc53 P b306efaa85e34d228d4ecee71b794136: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-3-root/wals/a7b072f91e494d8b995f146c2a9fdc53
I20251026 08:54:12.601565 7068 catalog_manager.cc:4985] TS b306efaa85e34d228d4ecee71b794136 (127.0.103.68:36793): tablet a7b072f91e494d8b995f146c2a9fdc53 (table test-workload [id=37afc5f2418743bb9dbae227c4793b5f]) successfully deleted
I20251026 08:54:12.854686 7659 ts_tablet_manager.cc:933] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Initiating tablet copy from peer 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511)
I20251026 08:54:12.855470 7659 tablet_copy_client.cc:323] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.66:41511
I20251026 08:54:12.856052 7268 tablet_copy_service.cc:140] P 618784080c4148edb8e4a00574b8d60a: Received BeginTabletCopySession request for tablet 8f780788b372466bbc705743cd502bba from peer b306efaa85e34d228d4ecee71b794136 ({username='slave'} at 127.0.0.1:60836)
I20251026 08:54:12.856164 7268 tablet_copy_service.cc:161] P 618784080c4148edb8e4a00574b8d60a: Beginning new tablet copy session on tablet 8f780788b372466bbc705743cd502bba from peer b306efaa85e34d228d4ecee71b794136 at {username='slave'} at 127.0.0.1:60836: session id = b306efaa85e34d228d4ecee71b794136-8f780788b372466bbc705743cd502bba
I20251026 08:54:12.858357 7268 tablet_copy_source_session.cc:215] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:12.859202 7659 ts_tablet_manager.cc:1916] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Deleting tablet data with delete state TABLET_DATA_COPYING
I20251026 08:54:12.864307 7659 ts_tablet_manager.cc:1929] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.2
I20251026 08:54:12.864519 7659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f780788b372466bbc705743cd502bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:12.867007 7659 tablet_copy_client.cc:806] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:12.867244 7659 tablet_copy_client.cc:670] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:12.868413 7659 tablet_copy_client.cc:538] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:12.871245 7659 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Bootstrap starting.
I20251026 08:54:12.877169 7659 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:12.877429 7659 tablet_bootstrap.cc:492] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Bootstrap complete.
I20251026 08:54:12.877612 7659 ts_tablet_manager.cc:1403] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251026 08:54:12.878224 7659 raft_consensus.cc:359] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.878389 7659 raft_consensus.cc:740] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b306efaa85e34d228d4ecee71b794136, State: Initialized, Role: LEARNER
I20251026 08:54:12.878553 7659 consensus_queue.cc:260] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.879041 7659 ts_tablet_manager.cc:1434] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20251026 08:54:12.879535 7268 tablet_copy_service.cc:342] P 618784080c4148edb8e4a00574b8d60a: Request end of tablet copy session b306efaa85e34d228d4ecee71b794136-8f780788b372466bbc705743cd502bba received from {username='slave'} at 127.0.0.1:60836
I20251026 08:54:12.879617 7268 tablet_copy_service.cc:434] P 618784080c4148edb8e4a00574b8d60a: ending tablet copy session b306efaa85e34d228d4ecee71b794136-8f780788b372466bbc705743cd502bba on tablet 8f780788b372466bbc705743cd502bba with peer b306efaa85e34d228d4ecee71b794136
I20251026 08:54:12.935807 7659 ts_tablet_manager.cc:933] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: Initiating tablet copy from peer b69381bade8d49e8802175e17fa72440 (127.0.103.67:44829)
I20251026 08:54:12.936553 7659 tablet_copy_client.cc:323] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: tablet copy: Beginning tablet copy session from remote peer at address 127.0.103.67:44829
I20251026 08:54:12.941758 7342 tablet_copy_service.cc:140] P b69381bade8d49e8802175e17fa72440: Received BeginTabletCopySession request for tablet 95d18ab1a92641e3bde0717dcc998271 from peer b306efaa85e34d228d4ecee71b794136 ({username='slave'} at 127.0.0.1:37464)
I20251026 08:54:12.941893 7342 tablet_copy_service.cc:161] P b69381bade8d49e8802175e17fa72440: Beginning new tablet copy session on tablet 95d18ab1a92641e3bde0717dcc998271 from peer b306efaa85e34d228d4ecee71b794136 at {username='slave'} at 127.0.0.1:37464: session id = b306efaa85e34d228d4ecee71b794136-95d18ab1a92641e3bde0717dcc998271
I20251026 08:54:12.943960 7342 tablet_copy_source_session.cc:215] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: Tablet Copy: opened 0 blocks and 1 log segments
I20251026 08:54:12.944988 7659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95d18ab1a92641e3bde0717dcc998271. 1 dirs total, 0 dirs full, 0 dirs failed
I20251026 08:54:12.949926 7659 tablet_copy_client.cc:806] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 0 data blocks...
I20251026 08:54:12.950174 7659 tablet_copy_client.cc:670] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: tablet copy: Starting download of 1 WAL segments...
I20251026 08:54:12.951411 7659 tablet_copy_client.cc:538] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251026 08:54:12.953771 7659 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: Bootstrap starting.
I20251026 08:54:12.959720 7659 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251026 08:54:12.960064 7659 tablet_bootstrap.cc:492] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: Bootstrap complete.
I20251026 08:54:12.960266 7659 ts_tablet_manager.cc:1403] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20251026 08:54:12.960865 7659 raft_consensus.cc:359] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.961045 7659 raft_consensus.cc:740] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b306efaa85e34d228d4ecee71b794136, State: Initialized, Role: LEARNER
I20251026 08:54:12.961236 7659 consensus_queue.cc:260] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: NON_VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: true } }
I20251026 08:54:12.961656 7659 ts_tablet_manager.cc:1434] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251026 08:54:12.962260 7342 tablet_copy_service.cc:342] P b69381bade8d49e8802175e17fa72440: Request end of tablet copy session b306efaa85e34d228d4ecee71b794136-95d18ab1a92641e3bde0717dcc998271 received from {username='slave'} at 127.0.0.1:37464
I20251026 08:54:12.962356 7342 tablet_copy_service.cc:434] P b69381bade8d49e8802175e17fa72440: ending tablet copy session b306efaa85e34d228d4ecee71b794136-95d18ab1a92641e3bde0717dcc998271 on tablet 95d18ab1a92641e3bde0717dcc998271 with peer b306efaa85e34d228d4ecee71b794136
I20251026 08:54:13.339341 7510 raft_consensus.cc:1217] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20251026 08:54:13.385749 7510 raft_consensus.cc:1217] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20251026 08:54:13.790318 7667 raft_consensus.cc:1064] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: attempting to promote NON_VOTER b306efaa85e34d228d4ecee71b794136 to VOTER
I20251026 08:54:13.791055 7667 consensus_queue.cc:237] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } }
I20251026 08:54:13.793027 7594 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 08:54:13.793027 7510 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 08:54:13.793093 7258 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 08:54:13.793614 7393 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:54:13.793820 7666 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:54:13.794250 7666 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:54:13.797261 7666 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.797866 7258 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.804175 7594 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.804286 7081 catalog_manager.cc:5649] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a reported cstate change: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.811151 7510 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.817168 7331 consensus_queue.cc:237] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } }
I20251026 08:54:13.818447 7510 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 08:54:13.818553 7594 raft_consensus.cc:1275] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer b69381bade8d49e8802175e17fa72440: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 08:54:13.818833 7671 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 08:54:13.819025 7666 consensus_queue.cc:1048] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 08:54:13.821166 7667 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 618784080c4148edb8e4a00574b8d60a (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.823459 7067 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 95d18ab1a92641e3bde0717dcc998271 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:54:13.824018 7510 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 618784080c4148edb8e4a00574b8d60a (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.824081 7594 raft_consensus.cc:2955] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 618784080c4148edb8e4a00574b8d60a (127.0.103.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.824353 7078 catalog_manager.cc:5649] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 reported cstate change: config changed from index 5 to 6, VOTER 618784080c4148edb8e4a00574b8d60a (127.0.103.66) evicted. New cstate: current_term: 1 leader_uuid: "b69381bade8d49e8802175e17fa72440" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:54:13.827785 7245 tablet_service.cc:1552] Processing DeleteTablet for tablet 95d18ab1a92641e3bde0717dcc998271 with delete_type TABLET_DATA_TOMBSTONED (TS 618784080c4148edb8e4a00574b8d60a not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:60806
I20251026 08:54:13.829928 7680 tablet_replica.cc:333] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:13.830104 7680 raft_consensus.cc:2243] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:13.830359 7680 raft_consensus.cc:2272] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:13.831036 7680 ts_tablet_manager.cc:1916] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:54:13.835188 7680 ts_tablet_manager.cc:1929] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20251026 08:54:13.835289 7680 log.cc:1199] T 95d18ab1a92641e3bde0717dcc998271 P 618784080c4148edb8e4a00574b8d60a: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-1-root/wals/95d18ab1a92641e3bde0717dcc998271
I20251026 08:54:13.835939 7066 catalog_manager.cc:4985] TS 618784080c4148edb8e4a00574b8d60a (127.0.103.66:41511): tablet 95d18ab1a92641e3bde0717dcc998271 (table test-workload [id=37afc5f2418743bb9dbae227c4793b5f]) successfully deleted
I20251026 08:54:13.882722 7668 raft_consensus.cc:1064] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: attempting to promote NON_VOTER b306efaa85e34d228d4ecee71b794136 to VOTER
I20251026 08:54:13.883308 7668 consensus_queue.cc:237] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } }
I20251026 08:54:13.885360 7331 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 08:54:13.885448 7510 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 LEARNER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 08:54:13.885885 7648 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:54:13.885916 7594 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251026 08:54:13.886339 7669 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:54:13.886529 7648 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251026 08:54:13.899408 7648 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.900053 7594 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.902590 7331 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.903425 7079 catalog_manager.cc:5649] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a reported cstate change: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20251026 08:54:13.903985 7509 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b306efaa85e34d228d4ecee71b794136 (127.0.103.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "16a1439266f547fcb516c394c7025029" member_type: VOTER last_known_addr { host: "127.0.103.65" port: 41911 } attrs { replace: true } } peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.909556 7258 consensus_queue.cc:237] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } }
I20251026 08:54:13.911337 7510 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 08:54:13.911865 7682 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 08:54:13.913518 7668 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 16a1439266f547fcb516c394c7025029 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.914091 7510 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 16a1439266f547fcb516c394c7025029 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.916815 7066 catalog_manager.cc:5162] ChangeConfig:REMOVE_PEER RPC for tablet 8f780788b372466bbc705743cd502bba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20251026 08:54:13.917886 7079 catalog_manager.cc:5649] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a reported cstate change: config changed from index 5 to 6, VOTER 16a1439266f547fcb516c394c7025029 (127.0.103.65) evicted. New cstate: current_term: 1 leader_uuid: "618784080c4148edb8e4a00574b8d60a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20251026 08:54:13.918439 7332 raft_consensus.cc:1275] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Refusing update from remote peer 618784080c4148edb8e4a00574b8d60a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251026 08:54:13.919025 7668 consensus_queue.cc:1048] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251026 08:54:13.920043 7332 raft_consensus.cc:2955] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 16a1439266f547fcb516c394c7025029 (127.0.103.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "618784080c4148edb8e4a00574b8d60a" member_type: VOTER last_known_addr { host: "127.0.103.66" port: 41511 } } peers { permanent_uuid: "b69381bade8d49e8802175e17fa72440" member_type: VOTER last_known_addr { host: "127.0.103.67" port: 44829 } } peers { permanent_uuid: "b306efaa85e34d228d4ecee71b794136" member_type: VOTER last_known_addr { host: "127.0.103.68" port: 36793 } attrs { promote: false } } }
I20251026 08:54:13.920742 7586 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f780788b372466bbc705743cd502bba with delete_type TABLET_DATA_TOMBSTONED (TS 16a1439266f547fcb516c394c7025029 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:53786
I20251026 08:54:13.921811 7686 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:13.921939 7686 raft_consensus.cc:2243] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:13.922220 7686 raft_consensus.cc:2272] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:13.923331 7686 ts_tablet_manager.cc:1916] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20251026 08:54:13.927047 7686 ts_tablet_manager.cc:1929] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20251026 08:54:13.927115 7686 log.cc:1199] T 8f780788b372466bbc705743cd502bba P 16a1439266f547fcb516c394c7025029: Deleting WAL directory at /tmp/dist-test-taskzY8M2O/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1761468669168814-413-0/minicluster-data/ts-0-root/wals/8f780788b372466bbc705743cd502bba
I20251026 08:54:13.927562 7068 catalog_manager.cc:4985] TS 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): tablet 8f780788b372466bbc705743cd502bba (table test-workload [id=37afc5f2418743bb9dbae227c4793b5f]) successfully deleted
W20251026 08:54:14.919049 7129 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver b306efaa85e34d228d4ecee71b794136 not available for placement
I20251026 08:54:15.554077 413 tablet_server.cc:178] TabletServer@127.0.103.65:41911 shutting down...
I20251026 08:54:15.567701 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:15.567979 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:15.568126 413 raft_consensus.cc:2243] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.568305 413 raft_consensus.cc:2272] T 86e6554eb93645fe95124508aa0a4171 P 16a1439266f547fcb516c394c7025029 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.568828 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:15.568966 413 raft_consensus.cc:2243] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.569147 413 raft_consensus.cc:2272] T 95d18ab1a92641e3bde0717dcc998271 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.569553 413 tablet_replica.cc:333] stopping tablet replica
I20251026 08:54:15.569674 413 raft_consensus.cc:2243] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.569832 413 raft_consensus.cc:2272] T a7b072f91e494d8b995f146c2a9fdc53 P 16a1439266f547fcb516c394c7025029 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251026 08:54:15.582450 7224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111) [suppressed 27 similar messages]
I20251026 08:54:15.583496 413 tablet_server.cc:195] TabletServer@127.0.103.65:41911 shutdown complete.
W20251026 08:54:15.583513 7224 consensus_peers.cc:597] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:15.587337 413 tablet_server.cc:178] TabletServer@127.0.103.66:0 shutting down...
I20251026 08:54:15.596024 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:15.596271 413 tablet_replica.cc:333] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:15.596454 413 raft_consensus.cc:2243] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:15.596716 413 raft_consensus.cc:2272] T 8f780788b372466bbc705743cd502bba P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.597241 413 tablet_replica.cc:333] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:15.597357 413 raft_consensus.cc:2243] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:15.597569 413 raft_consensus.cc:2272] T a7b072f91e494d8b995f146c2a9fdc53 P 618784080c4148edb8e4a00574b8d60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.597980 413 tablet_replica.cc:333] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a: stopping tablet replica
I20251026 08:54:15.598145 413 raft_consensus.cc:2243] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 2 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.598325 413 raft_consensus.cc:2272] T 86e6554eb93645fe95124508aa0a4171 P 618784080c4148edb8e4a00574b8d60a [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.613320 413 tablet_server.cc:195] TabletServer@127.0.103.66:0 shutdown complete.
W20251026 08:54:15.616720 7298 consensus_peers.cc:597] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 -> Peer 16a1439266f547fcb516c394c7025029 (127.0.103.65:41911): Couldn't send request to peer 16a1439266f547fcb516c394c7025029. Status: Network error: Client connection negotiation failed: client connection to 127.0.103.65:41911: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251026 08:54:15.617414 413 tablet_server.cc:178] TabletServer@127.0.103.67:0 shutting down...
I20251026 08:54:15.635582 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:15.635818 413 tablet_replica.cc:333] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:15.635991 413 raft_consensus.cc:2243] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:15.636274 413 raft_consensus.cc:2272] T 95d18ab1a92641e3bde0717dcc998271 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.636803 413 tablet_replica.cc:333] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:15.636929 413 raft_consensus.cc:2243] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 LEADER]: Raft consensus shutting down.
I20251026 08:54:15.637149 413 raft_consensus.cc:2272] T 86e6554eb93645fe95124508aa0a4171 P b69381bade8d49e8802175e17fa72440 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.637542 413 tablet_replica.cc:333] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:15.637653 413 raft_consensus.cc:2243] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.637869 413 raft_consensus.cc:2272] T 8f780788b372466bbc705743cd502bba P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.638312 413 tablet_replica.cc:333] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440: stopping tablet replica
I20251026 08:54:15.638402 413 raft_consensus.cc:2243] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.638553 413 raft_consensus.cc:2272] T a7b072f91e494d8b995f146c2a9fdc53 P b69381bade8d49e8802175e17fa72440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.652151 413 tablet_server.cc:195] TabletServer@127.0.103.67:0 shutdown complete.
I20251026 08:54:15.655968 413 tablet_server.cc:178] TabletServer@127.0.103.68:0 shutting down...
I20251026 08:54:15.664352 413 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20251026 08:54:15.664537 413 tablet_replica.cc:333] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136: stopping tablet replica
I20251026 08:54:15.664703 413 raft_consensus.cc:2243] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.664885 413 raft_consensus.cc:2272] T 95d18ab1a92641e3bde0717dcc998271 P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.665397 413 tablet_replica.cc:333] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136: stopping tablet replica
I20251026 08:54:15.665571 413 raft_consensus.cc:2243] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251026 08:54:15.665795 413 raft_consensus.cc:2272] T 8f780788b372466bbc705743cd502bba P b306efaa85e34d228d4ecee71b794136 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.679937 413 tablet_server.cc:195] TabletServer@127.0.103.68:0 shutdown complete.
I20251026 08:54:15.682845 413 master.cc:561] Master@127.0.103.126:36111 shutting down...
I20251026 08:54:15.689836 413 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 1 LEADER]: Raft consensus shutting down.
I20251026 08:54:15.690002 413 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251026 08:54:15.690120 413 tablet_replica.cc:333] T 00000000000000000000000000000000 P 546cf24eda40447097b89835df9094f8: stopping tablet replica
I20251026 08:54:15.702909 413 master.cc:583] Master@127.0.103.126:36111 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (7406 ms)
[----------] 14 tests from AutoRebalancerTest (186484 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (186485 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AutoRebalancerTest.TestMaxMovesPerServer
1 FAILED TEST
I20251026 08:54:15.715950 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5327: suppressed but not reported on 6 messages since previous log ~4 seconds ago
I20251026 08:54:15.716006 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20251026 08:54:15.716050 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20251026 08:54:15.716085 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4712: suppressed but not reported on 11 messages since previous log ~4 seconds ago
I20251026 08:54:15.716126 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 8 messages since previous log ~172 seconds ago
I20251026 08:54:15.716198 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 50 messages since previous log ~17 seconds ago
I20251026 08:54:15.716238 413 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 1 messages since previous log ~0 seconds ago