[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from AutoRebalancerTest
[ RUN ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250905 02:20:20.907035 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:46065,127.31.48.189:38549,127.31.48.188:37595
I20250905 02:20:20.908771 31938 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 02:20:20.909770 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:20.927924 31945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:20.927937 31947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:20.930400 31938 server_base.cc:1047] running on GCE node
W20250905 02:20:20.929029 31944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:22.089613 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:22.089852 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:22.090045 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038822090019 us; error 0 us; skew 500 ppm
I20250905 02:20:22.090857 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:22.096340 31938 webserver.cc:480] Webserver started at http://127.31.48.190:45225/ using document root <none> and password file <none>
I20250905 02:20:22.097216 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:22.097419 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:22.097875 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:22.103518 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "61f391b1c201478fbe79121f00d8300a"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.111059 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250905 02:20:22.115676 31953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:22.116667 31938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 02:20:22.117007 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "61f391b1c201478fbe79121f00d8300a"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.117326 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:22.167837 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:22.169232 31938 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 02:20:22.169639 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:22.246409 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:46065
I20250905 02:20:22.246457 32004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:46065 every 8 connection(s)
I20250905 02:20:22.253170 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:22.254989 32005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 02:20:22.259985 32007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:22.261683 32008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:22.265857 32010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:22.266017 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:22.266902 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:22.267158 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:22.267359 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038822267341 us; error 0 us; skew 500 ppm
I20250905 02:20:22.268119 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:22.272547 31938 webserver.cc:480] Webserver started at http://127.31.48.189:45055/ using document root <none> and password file <none>
I20250905 02:20:22.273213 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:22.273469 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:22.273815 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:22.275202 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root/instance:
uuid: "41834a961e8d48afa8fbb1c7e060bdef"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.281963 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.008s
I20250905 02:20:22.279981 32005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:22.289183 32015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:22.290987 31938 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20250905 02:20:22.291311 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root
uuid: "41834a961e8d48afa8fbb1c7e060bdef"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.291682 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:22.311040 32005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:22.313220 31955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.189:38549: connect: Connection refused (error 111)
W20250905 02:20:22.317639 32005 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.189:38549: Network error: Client connection negotiation failed: client connection to 127.31.48.189:38549: connect: Connection refused (error 111)
I20250905 02:20:22.318912 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:22.320070 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:22.367143 32005 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } attempt: 1
W20250905 02:20:22.372593 32005 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.189:38549: Network error: Client connection negotiation failed: client connection to 127.31.48.189:38549: connect: Connection refused (error 111)
I20250905 02:20:22.394650 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.189:38549
I20250905 02:20:22.394830 32069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.189:38549 every 8 connection(s)
I20250905 02:20:22.400020 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:22.400660 32070 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 02:20:22.407330 32072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:22.408068 32070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:22.410849 32073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:22.412948 32075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:22.416718 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:22.417502 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:22.417693 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:22.417858 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038822417841 us; error 0 us; skew 500 ppm
I20250905 02:20:22.418349 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:22.421692 31938 webserver.cc:480] Webserver started at http://127.31.48.188:41641/ using document root <none> and password file <none>
I20250905 02:20:22.422268 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:22.422501 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:22.423218 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:22.424211 32070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:22.424419 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root/instance:
uuid: "c375e6a5c0a149139fd24f1f79497fa4"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.430490 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 02:20:22.435451 32082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:22.436209 31938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 02:20:22.436523 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root
uuid: "c375e6a5c0a149139fd24f1f79497fa4"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.436794 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:22.441546 32005 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } attempt: 2
I20250905 02:20:22.441617 32070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:22.450368 32070 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:37595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:37595: connect: Connection refused (error 111)
I20250905 02:20:22.454363 32005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:22.458714 32005 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:37595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:37595: connect: Connection refused (error 111)
I20250905 02:20:22.459492 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:22.460755 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:22.490437 32005 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } attempt: 1
I20250905 02:20:22.493764 32070 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } attempt: 1
W20250905 02:20:22.494796 32005 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:37595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:37595: connect: Connection refused (error 111)
W20250905 02:20:22.497771 32070 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:37595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:37595: connect: Connection refused (error 111)
I20250905 02:20:22.535633 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.188:37595
I20250905 02:20:22.535739 32134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.188:37595 every 8 connection(s)
I20250905 02:20:22.539785 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:20:22.541172 32135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:22.546056 32135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:22.557375 32135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:22.565146 32070 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } attempt: 2
I20250905 02:20:22.565611 32135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:22.570386 32005 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } attempt: 2
I20250905 02:20:22.590741 32005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: Bootstrap starting.
I20250905 02:20:22.590749 32135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: Bootstrap starting.
I20250905 02:20:22.590780 32070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: Bootstrap starting.
I20250905 02:20:22.598492 32135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:22.598923 32005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:22.599256 32070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:22.601634 32135 log.cc:826] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: Log is configured to *not* fsync() on all Append() calls
I20250905 02:20:22.609552 32005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: No bootstrap required, opened a new log
I20250905 02:20:22.609709 32135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: No bootstrap required, opened a new log
I20250905 02:20:22.609990 32070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: No bootstrap required, opened a new log
I20250905 02:20:22.627677 32005 raft_consensus.cc:357] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.627694 32070 raft_consensus.cc:357] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.627694 32135 raft_consensus.cc:357] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.628417 32005 raft_consensus.cc:383] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:22.628688 32070 raft_consensus.cc:383] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:22.628786 32135 raft_consensus.cc:383] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:22.628846 32005 raft_consensus.cc:738] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f391b1c201478fbe79121f00d8300a, State: Initialized, Role: FOLLOWER
I20250905 02:20:22.629033 32070 raft_consensus.cc:738] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41834a961e8d48afa8fbb1c7e060bdef, State: Initialized, Role: FOLLOWER
I20250905 02:20:22.629197 32135 raft_consensus.cc:738] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c375e6a5c0a149139fd24f1f79497fa4, State: Initialized, Role: FOLLOWER
I20250905 02:20:22.629945 32135 consensus_queue.cc:260] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.629945 32005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.629935 32070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.634893 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.635509 32005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:22.635665 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:22.636808 32145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.637816 32145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:22.637492 32135 sys_catalog.cc:564] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:22.638110 32070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:22.638824 32144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.639585 32144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: This master's current role is: FOLLOWER
W20250905 02:20:22.674989 31959 tablet.cc:2378] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:20:22.685463 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250905 02:20:22.684543 32168 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250905 02:20:22.699198 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250905 02:20:22.701485 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:22.706166 32179 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 02:20:22.706516 32179 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250905 02:20:22.707468 32178 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 02:20:22.707777 32178 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250905 02:20:22.712637 32181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:22.714293 32182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:22.722863 32184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:22.723400 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:22.724400 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:22.724668 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:22.724867 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038822724851 us; error 0 us; skew 500 ppm
I20250905 02:20:22.725518 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:22.728441 31938 webserver.cc:480] Webserver started at http://127.31.48.129:42923/ using document root <none> and password file <none>
I20250905 02:20:22.729100 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:22.729341 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:22.729359 32146 raft_consensus.cc:491] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:22.729831 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:22.730086 32146 raft_consensus.cc:513] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.731364 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "c9a93c7b310c41dc8d8aa180da8994b8"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.733529 32146 leader_election.cc:290] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61f391b1c201478fbe79121f00d8300a (127.31.48.190:46065), 41834a961e8d48afa8fbb1c7e060bdef (127.31.48.189:38549)
I20250905 02:20:22.733613 31980 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c375e6a5c0a149139fd24f1f79497fa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f391b1c201478fbe79121f00d8300a" is_pre_election: true
I20250905 02:20:22.734303 31980 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c375e6a5c0a149139fd24f1f79497fa4 in term 0.
I20250905 02:20:22.735100 32045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c375e6a5c0a149139fd24f1f79497fa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41834a961e8d48afa8fbb1c7e060bdef" is_pre_election: true
I20250905 02:20:22.735536 32083 leader_election.cc:304] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f391b1c201478fbe79121f00d8300a, c375e6a5c0a149139fd24f1f79497fa4; no voters:
I20250905 02:20:22.735760 32045 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c375e6a5c0a149139fd24f1f79497fa4 in term 0.
I20250905 02:20:22.736452 32146 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:22.736860 32146 raft_consensus.cc:491] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:22.737205 32146 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:22.742550 32146 raft_consensus.cc:513] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.744987 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.013s user 0.009s sys 0.004s
I20250905 02:20:22.745115 31980 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c375e6a5c0a149139fd24f1f79497fa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61f391b1c201478fbe79121f00d8300a"
I20250905 02:20:22.745474 32045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c375e6a5c0a149139fd24f1f79497fa4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41834a961e8d48afa8fbb1c7e060bdef"
I20250905 02:20:22.745693 31980 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:22.746021 32045 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:22.753175 31980 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c375e6a5c0a149139fd24f1f79497fa4 in term 1.
I20250905 02:20:22.754382 32083 leader_election.cc:304] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61f391b1c201478fbe79121f00d8300a, c375e6a5c0a149139fd24f1f79497fa4; no voters:
I20250905 02:20:22.755106 32045 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c375e6a5c0a149139fd24f1f79497fa4 in term 1.
I20250905 02:20:22.756656 32146 leader_election.cc:290] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [CANDIDATE]: Term 1 election: Requested vote from peers 61f391b1c201478fbe79121f00d8300a (127.31.48.190:46065), 41834a961e8d48afa8fbb1c7e060bdef (127.31.48.189:38549)
I20250905 02:20:22.757414 32146 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:22.758011 32146 raft_consensus.cc:695] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 1 LEADER]: Becoming Leader. State: Replica: c375e6a5c0a149139fd24f1f79497fa4, State: Running, Role: LEADER
I20250905 02:20:22.759289 32146 consensus_queue.cc:237] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } }
I20250905 02:20:22.762624 32189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:22.763612 31938 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.010s sys 0.005s
I20250905 02:20:22.764057 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "c9a93c7b310c41dc8d8aa180da8994b8"
format_stamp: "Formatted at 2025-09-05 02:20:22 on dist-test-slave-q32q"
I20250905 02:20:22.764532 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:22.775719 32190 sys_catalog.cc:455] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c375e6a5c0a149139fd24f1f79497fa4. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.776628 32190 sys_catalog.cc:458] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:22.779520 32196 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:22.783944 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:22.786056 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:22.787339 32196 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:22.787755 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:22.792610 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:22.792868 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:22.793148 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:22.793346 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:22.807910 32045 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 1 FOLLOWER]: Refusing update from remote peer c375e6a5c0a149139fd24f1f79497fa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:22.808820 31980 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 1 FOLLOWER]: Refusing update from remote peer c375e6a5c0a149139fd24f1f79497fa4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:22.809770 32146 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:22.811033 32190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:22.830024 32144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c375e6a5c0a149139fd24f1f79497fa4. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.831293 32144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:22.847529 32145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: SysCatalogTable state changed. Reason: New leader c375e6a5c0a149139fd24f1f79497fa4. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.848297 32145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:22.850445 32200 mvcc.cc:204] Tried to move back new op lower bound from 7196831018197204992 to 7196831018035171328. Current Snapshot: MvccSnapshot[applied={T|T < 7196831018197204992}]
I20250905 02:20:22.871706 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.873868 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:22.877408 32144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.878351 32144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:22.880282 32196 catalog_manager.cc:1349] Generated new cluster ID: 23a7484851334019861fd4b2c1c4e260
I20250905 02:20:22.880692 32196 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:22.895428 32145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.896298 32145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:22.895946 32146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c375e6a5c0a149139fd24f1f79497fa4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61f391b1c201478fbe79121f00d8300a" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 46065 } } peers { permanent_uuid: "41834a961e8d48afa8fbb1c7e060bdef" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38549 } } peers { permanent_uuid: "c375e6a5c0a149139fd24f1f79497fa4" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 37595 } } }
I20250905 02:20:22.896667 32146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:22.925423 32196 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:22.927556 32196 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:22.971378 32196 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: Generated new TSK 0
I20250905 02:20:22.972568 32196 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:23.037426 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:38163
I20250905 02:20:23.038175 32263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:38163 every 8 connection(s)
I20250905 02:20:23.060366 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:23.067889 32264 heartbeater.cc:344] Connected to a master server at 127.31.48.188:37595
I20250905 02:20:23.068595 32264 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.070137 32264 heartbeater.cc:507] Master 127.31.48.188:37595 requested a full tablet report, sending...
I20250905 02:20:23.074841 32100 ts_manager.cc:194] Registered new tserver with Master: c9a93c7b310c41dc8d8aa180da8994b8 (127.31.48.129:38163)
I20250905 02:20:23.078979 32266 heartbeater.cc:344] Connected to a master server at 127.31.48.190:46065
I20250905 02:20:23.079519 32266 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.080539 32266 heartbeater.cc:507] Master 127.31.48.190:46065 requested a full tablet report, sending...
I20250905 02:20:23.081328 32100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43568
I20250905 02:20:23.084277 31970 ts_manager.cc:194] Registered new tserver with Master: c9a93c7b310c41dc8d8aa180da8994b8 (127.31.48.129:38163)
W20250905 02:20:23.089581 32271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:23.092638 32272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:23.100450 32265 heartbeater.cc:344] Connected to a master server at 127.31.48.189:38549
I20250905 02:20:23.100927 32265 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.101612 32265 heartbeater.cc:507] Master 127.31.48.189:38549 requested a full tablet report, sending...
W20250905 02:20:23.103847 32274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:23.104072 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:23.104837 32035 ts_manager.cc:194] Registered new tserver with Master: c9a93c7b310c41dc8d8aa180da8994b8 (127.31.48.129:38163)
I20250905 02:20:23.105326 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:23.105602 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:23.105814 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038823105796 us; error 0 us; skew 500 ppm
I20250905 02:20:23.106518 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:23.109463 31938 webserver.cc:480] Webserver started at http://127.31.48.130:38625/ using document root <none> and password file <none>
I20250905 02:20:23.110080 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:23.110329 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:23.110654 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:23.112069 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "89a4b15d6796461880bcb176d3778333"
format_stamp: "Formatted at 2025-09-05 02:20:23 on dist-test-slave-q32q"
I20250905 02:20:23.117774 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 02:20:23.121850 32279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:23.122593 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20250905 02:20:23.122908 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "89a4b15d6796461880bcb176d3778333"
format_stamp: "Formatted at 2025-09-05 02:20:23 on dist-test-slave-q32q"
I20250905 02:20:23.123221 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:23.141327 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:23.142756 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:23.144596 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:23.148093 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:23.148377 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:23.148651 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:23.148867 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:23.243624 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:36217
I20250905 02:20:23.244513 32341 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:36217 every 8 connection(s)
I20250905 02:20:23.268704 32342 heartbeater.cc:344] Connected to a master server at 127.31.48.188:37595
I20250905 02:20:23.269168 32342 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.270167 32342 heartbeater.cc:507] Master 127.31.48.188:37595 requested a full tablet report, sending...
I20250905 02:20:23.272716 32100 ts_manager.cc:194] Registered new tserver with Master: 89a4b15d6796461880bcb176d3778333 (127.31.48.130:36217)
I20250905 02:20:23.275070 32100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43570
I20250905 02:20:23.277870 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:23.277916 32343 heartbeater.cc:344] Connected to a master server at 127.31.48.189:38549
I20250905 02:20:23.278472 32343 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.279716 32343 heartbeater.cc:507] Master 127.31.48.189:38549 requested a full tablet report, sending...
I20250905 02:20:23.282778 32035 ts_manager.cc:194] Registered new tserver with Master: 89a4b15d6796461880bcb176d3778333 (127.31.48.130:36217)
I20250905 02:20:23.286164 32345 heartbeater.cc:344] Connected to a master server at 127.31.48.190:46065
I20250905 02:20:23.286540 32345 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.287581 32345 heartbeater.cc:507] Master 127.31.48.190:46065 requested a full tablet report, sending...
W20250905 02:20:23.288810 32349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:23.290493 31970 ts_manager.cc:194] Registered new tserver with Master: 89a4b15d6796461880bcb176d3778333 (127.31.48.130:36217)
W20250905 02:20:23.291541 32350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:23.296278 31938 server_base.cc:1047] running on GCE node
W20250905 02:20:23.297016 32352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:23.297986 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:23.298256 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:23.298468 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038823298451 us; error 0 us; skew 500 ppm
I20250905 02:20:23.299115 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:23.301923 31938 webserver.cc:480] Webserver started at http://127.31.48.131:36535/ using document root <none> and password file <none>
I20250905 02:20:23.302522 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:23.302765 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:23.303123 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:23.304585 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "96447898b47c4c13a25b9bacc29a665b"
format_stamp: "Formatted at 2025-09-05 02:20:23 on dist-test-slave-q32q"
I20250905 02:20:23.310534 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250905 02:20:23.314702 32357 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:23.315517 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:20:23.315846 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "96447898b47c4c13a25b9bacc29a665b"
format_stamp: "Formatted at 2025-09-05 02:20:23 on dist-test-slave-q32q"
I20250905 02:20:23.316170 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:23.342438 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:23.344045 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:23.346364 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:23.350093 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:23.350353 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:23.350647 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:23.350860 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:23.458184 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:40765
I20250905 02:20:23.458325 32419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:40765 every 8 connection(s)
I20250905 02:20:23.492022 32420 heartbeater.cc:344] Connected to a master server at 127.31.48.188:37595
I20250905 02:20:23.492475 32420 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.493484 32420 heartbeater.cc:507] Master 127.31.48.188:37595 requested a full tablet report, sending...
I20250905 02:20:23.494971 32421 heartbeater.cc:344] Connected to a master server at 127.31.48.189:38549
I20250905 02:20:23.495349 32421 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.495791 32100 ts_manager.cc:194] Registered new tserver with Master: 96447898b47c4c13a25b9bacc29a665b (127.31.48.131:40765)
I20250905 02:20:23.496223 32421 heartbeater.cc:507] Master 127.31.48.189:38549 requested a full tablet report, sending...
I20250905 02:20:23.497503 32100 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:43576
I20250905 02:20:23.499099 32422 heartbeater.cc:344] Connected to a master server at 127.31.48.190:46065
I20250905 02:20:23.499495 32422 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:23.500002 32035 ts_manager.cc:194] Registered new tserver with Master: 96447898b47c4c13a25b9bacc29a665b (127.31.48.131:40765)
I20250905 02:20:23.500239 32422 heartbeater.cc:507] Master 127.31.48.190:46065 requested a full tablet report, sending...
I20250905 02:20:23.502635 31970 ts_manager.cc:194] Registered new tserver with Master: 96447898b47c4c13a25b9bacc29a665b (127.31.48.131:40765)
I20250905 02:20:23.503152 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01170234s
I20250905 02:20:23.690073 32168 catalog_manager.cc:1261] Loaded cluster ID: 23a7484851334019861fd4b2c1c4e260
I20250905 02:20:23.690371 32168 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: loading cluster ID for follower catalog manager: success
I20250905 02:20:23.694988 32168 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: acquiring CA information for follower catalog manager: success
I20250905 02:20:23.698366 32168 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 02:20:23.711592 32178 catalog_manager.cc:1261] Loaded cluster ID: 23a7484851334019861fd4b2c1c4e260
I20250905 02:20:23.711903 32178 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: loading cluster ID for follower catalog manager: success
I20250905 02:20:23.716228 32178 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: acquiring CA information for follower catalog manager: success
I20250905 02:20:23.719621 32178 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 02:20:24.087940 32264 heartbeater.cc:499] Master 127.31.48.188:37595 was elected leader, sending a full tablet report...
I20250905 02:20:24.278110 32342 heartbeater.cc:499] Master 127.31.48.188:37595 was elected leader, sending a full tablet report...
I20250905 02:20:24.500524 32420 heartbeater.cc:499] Master 127.31.48.188:37595 was elected leader, sending a full tablet report...
I20250905 02:20:24.698107 32177 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:24.809931 31938 test_util.cc:276] Using random seed: 2063754673
I20250905 02:20:24.868774 32100 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43586:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 02:20:24.961400 32385 tablet_service.cc:1468] Processing CreateTablet for tablet 569af7fb0d5746c0b4e9033398e2d251 (DEFAULT_TABLE table=test-workload [id=e4b24f3fe94b4a02889d29a9d9abc308]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:20:24.963549 32385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 569af7fb0d5746c0b4e9033398e2d251. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:24.964996 32229 tablet_service.cc:1468] Processing CreateTablet for tablet 0b628aec56ee46ce8d6056a2e31b2da3 (DEFAULT_TABLE table=test-workload [id=e4b24f3fe94b4a02889d29a9d9abc308]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:20:24.966676 32228 tablet_service.cc:1468] Processing CreateTablet for tablet 9b8df591e95e4347bd7dc0fa5bf6577b (DEFAULT_TABLE table=test-workload [id=e4b24f3fe94b4a02889d29a9d9abc308]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:20:24.967175 32229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b628aec56ee46ce8d6056a2e31b2da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:24.968021 32228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8df591e95e4347bd7dc0fa5bf6577b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:24.972741 32307 tablet_service.cc:1468] Processing CreateTablet for tablet 88f4a3754e2f4750bfe3de87f331282f (DEFAULT_TABLE table=test-workload [id=e4b24f3fe94b4a02889d29a9d9abc308]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:20:24.974568 32307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88f4a3754e2f4750bfe3de87f331282f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:24.994311 32455 tablet_bootstrap.cc:492] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b: Bootstrap starting.
I20250905 02:20:24.999687 32456 tablet_bootstrap.cc:492] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8: Bootstrap starting.
I20250905 02:20:25.002275 32457 tablet_bootstrap.cc:492] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333: Bootstrap starting.
I20250905 02:20:25.004753 32455 tablet_bootstrap.cc:654] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:25.010344 32456 tablet_bootstrap.cc:654] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:25.011024 32457 tablet_bootstrap.cc:654] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:25.015570 32455 tablet_bootstrap.cc:492] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b: No bootstrap required, opened a new log
I20250905 02:20:25.016213 32455 ts_tablet_manager.cc:1397] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.012s
I20250905 02:20:25.018674 32457 tablet_bootstrap.cc:492] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333: No bootstrap required, opened a new log
I20250905 02:20:25.019117 32457 ts_tablet_manager.cc:1397] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20250905 02:20:25.019002 32455 raft_consensus.cc:357] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96447898b47c4c13a25b9bacc29a665b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 40765 } }
I20250905 02:20:25.019870 32456 tablet_bootstrap.cc:492] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8: No bootstrap required, opened a new log
I20250905 02:20:25.020119 32455 raft_consensus.cc:383] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:25.020391 32456 ts_tablet_manager.cc:1397] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.012s
I20250905 02:20:25.020507 32455 raft_consensus.cc:738] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96447898b47c4c13a25b9bacc29a665b, State: Initialized, Role: FOLLOWER
I20250905 02:20:25.021261 32455 consensus_queue.cc:260] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96447898b47c4c13a25b9bacc29a665b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 40765 } }
I20250905 02:20:25.021909 32455 raft_consensus.cc:397] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:25.022064 32457 raft_consensus.cc:357] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a4b15d6796461880bcb176d3778333" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36217 } }
I20250905 02:20:25.022660 32457 raft_consensus.cc:383] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:25.022570 32456 raft_consensus.cc:357] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.022918 32457 raft_consensus.cc:738] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89a4b15d6796461880bcb176d3778333, State: Initialized, Role: FOLLOWER
I20250905 02:20:25.023058 32456 raft_consensus.cc:383] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:25.022272 32455 raft_consensus.cc:491] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:25.023360 32456 raft_consensus.cc:738] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a93c7b310c41dc8d8aa180da8994b8, State: Initialized, Role: FOLLOWER
I20250905 02:20:25.023475 32457 consensus_queue.cc:260] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a4b15d6796461880bcb176d3778333" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36217 } }
I20250905 02:20:25.024583 32455 raft_consensus.cc:3058] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:25.025027 32457 raft_consensus.cc:397] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:25.024932 32456 consensus_queue.cc:260] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.025445 32457 raft_consensus.cc:491] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:25.025682 32456 raft_consensus.cc:397] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:25.025866 32457 raft_consensus.cc:3058] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:25.026026 32456 raft_consensus.cc:491] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:25.026425 32456 raft_consensus.cc:3058] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:25.032658 32456 raft_consensus.cc:513] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.032594 32455 raft_consensus.cc:513] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96447898b47c4c13a25b9bacc29a665b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 40765 } }
I20250905 02:20:25.033063 32457 raft_consensus.cc:513] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a4b15d6796461880bcb176d3778333" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36217 } }
I20250905 02:20:25.033401 32456 leader_election.cc:304] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a93c7b310c41dc8d8aa180da8994b8; no voters:
I20250905 02:20:25.033545 32455 leader_election.cc:304] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96447898b47c4c13a25b9bacc29a665b; no voters:
I20250905 02:20:25.033763 32457 leader_election.cc:304] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89a4b15d6796461880bcb176d3778333; no voters:
I20250905 02:20:25.037250 32455 leader_election.cc:290] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:25.037885 32462 raft_consensus.cc:2802] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:25.040771 32456 leader_election.cc:290] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:25.041265 32461 raft_consensus.cc:2802] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:25.050223 32455 ts_tablet_manager.cc:1428] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b: Time spent starting tablet: real 0.034s user 0.007s sys 0.018s
I20250905 02:20:25.051995 32462 raft_consensus.cc:695] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 1 LEADER]: Becoming Leader. State: Replica: 96447898b47c4c13a25b9bacc29a665b, State: Running, Role: LEADER
I20250905 02:20:25.052820 32462 consensus_queue.cc:237] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96447898b47c4c13a25b9bacc29a665b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 40765 } }
I20250905 02:20:25.053373 32461 raft_consensus.cc:695] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 LEADER]: Becoming Leader. State: Replica: c9a93c7b310c41dc8d8aa180da8994b8, State: Running, Role: LEADER
I20250905 02:20:25.054411 32461 consensus_queue.cc:237] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.057740 32463 raft_consensus.cc:2802] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:25.062706 32457 leader_election.cc:290] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:25.090065 32463 raft_consensus.cc:695] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 1 LEADER]: Becoming Leader. State: Replica: 89a4b15d6796461880bcb176d3778333, State: Running, Role: LEADER
I20250905 02:20:25.107560 32100 catalog_manager.cc:5582] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a93c7b310c41dc8d8aa180da8994b8 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:25.109658 32099 catalog_manager.cc:5582] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b reported cstate change: term changed from 0 to 1, leader changed from <none> to 96447898b47c4c13a25b9bacc29a665b (127.31.48.131). New cstate: current_term: 1 leader_uuid: "96447898b47c4c13a25b9bacc29a665b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96447898b47c4c13a25b9bacc29a665b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 40765 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:25.114565 32456 ts_tablet_manager.cc:1428] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8: Time spent starting tablet: real 0.094s user 0.027s sys 0.047s
I20250905 02:20:25.099382 32463 consensus_queue.cc:237] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a4b15d6796461880bcb176d3778333" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36217 } }
I20250905 02:20:25.119513 32457 ts_tablet_manager.cc:1428] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333: Time spent starting tablet: real 0.100s user 0.025s sys 0.001s
I20250905 02:20:25.122464 32456 tablet_bootstrap.cc:492] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8: Bootstrap starting.
I20250905 02:20:25.137249 32456 tablet_bootstrap.cc:654] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:25.156850 32098 catalog_manager.cc:5582] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89a4b15d6796461880bcb176d3778333 (127.31.48.130). New cstate: current_term: 1 leader_uuid: "89a4b15d6796461880bcb176d3778333" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89a4b15d6796461880bcb176d3778333" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36217 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:25.197980 32456 tablet_bootstrap.cc:492] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8: No bootstrap required, opened a new log
I20250905 02:20:25.198596 32456 ts_tablet_manager.cc:1397] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8: Time spent bootstrapping tablet: real 0.076s user 0.034s sys 0.029s
I20250905 02:20:25.201380 32456 raft_consensus.cc:357] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.202536 32456 raft_consensus.cc:383] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:25.202881 32456 raft_consensus.cc:738] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a93c7b310c41dc8d8aa180da8994b8, State: Initialized, Role: FOLLOWER
I20250905 02:20:25.203943 32456 consensus_queue.cc:260] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.204721 32456 raft_consensus.cc:397] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:25.205000 32456 raft_consensus.cc:491] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:25.205338 32456 raft_consensus.cc:3058] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:25.212409 32456 raft_consensus.cc:513] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.213155 32456 leader_election.cc:304] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a93c7b310c41dc8d8aa180da8994b8; no voters:
I20250905 02:20:25.213749 32456 leader_election.cc:290] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:25.213882 32461 raft_consensus.cc:2802] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:25.214342 32461 raft_consensus.cc:695] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 LEADER]: Becoming Leader. State: Replica: c9a93c7b310c41dc8d8aa180da8994b8, State: Running, Role: LEADER
I20250905 02:20:25.215005 32461 consensus_queue.cc:237] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } }
I20250905 02:20:25.215456 32456 ts_tablet_manager.cc:1428] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20250905 02:20:25.224412 32098 catalog_manager.cc:5582] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9a93c7b310c41dc8d8aa180da8994b8 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a93c7b310c41dc8d8aa180da8994b8" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 38163 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:25.279035 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:20:25.298025 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:25.298811 31938 tablet_replica.cc:331] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8: stopping tablet replica
I20250905 02:20:25.299396 31938 raft_consensus.cc:2241] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:25.299860 31938 raft_consensus.cc:2270] T 0b628aec56ee46ce8d6056a2e31b2da3 P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.302480 31938 tablet_replica.cc:331] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8: stopping tablet replica
I20250905 02:20:25.303041 31938 raft_consensus.cc:2241] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:25.303416 31938 raft_consensus.cc:2270] T 9b8df591e95e4347bd7dc0fa5bf6577b P c9a93c7b310c41dc8d8aa180da8994b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.445031 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:20:25.453554 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:20:25.469152 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:25.469844 31938 tablet_replica.cc:331] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333: stopping tablet replica
I20250905 02:20:25.470307 31938 raft_consensus.cc:2241] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:25.470695 31938 raft_consensus.cc:2270] T 88f4a3754e2f4750bfe3de87f331282f P 89a4b15d6796461880bcb176d3778333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.489240 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:20:25.496315 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:20:25.510423 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:25.511092 31938 tablet_replica.cc:331] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b: stopping tablet replica
I20250905 02:20:25.511554 31938 raft_consensus.cc:2241] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:25.511969 31938 raft_consensus.cc:2270] T 569af7fb0d5746c0b4e9033398e2d251 P 96447898b47c4c13a25b9bacc29a665b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.530474 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:20:25.537917 31938 master.cc:561] Master@127.31.48.190:46065 shutting down...
I20250905 02:20:25.635564 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:25.636067 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.636416 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 61f391b1c201478fbe79121f00d8300a: stopping tablet replica
I20250905 02:20:25.654934 31938 master.cc:583] Master@127.31.48.190:46065 shutdown complete.
I20250905 02:20:25.664274 31938 master.cc:561] Master@127.31.48.189:38549 shutting down...
I20250905 02:20:25.676688 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:25.677255 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.677521 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 41834a961e8d48afa8fbb1c7e060bdef: stopping tablet replica
I20250905 02:20:25.716635 31938 master.cc:583] Master@127.31.48.189:38549 shutdown complete.
I20250905 02:20:25.727731 31938 master.cc:561] Master@127.31.48.188:37595 shutting down...
I20250905 02:20:25.742098 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:25.743032 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:25.743346 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P c375e6a5c0a149139fd24f1f79497fa4: stopping tablet replica
I20250905 02:20:25.761639 31938 master.cc:583] Master@127.31.48.188:37595 shutdown complete.
[ OK ] AutoRebalancerTest.OnlyLeaderDoesAutoRebalancing (4883 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn
I20250905 02:20:25.786557 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:39045
I20250905 02:20:25.787577 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:25.791848 32478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:25.792784 32479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:25.794257 31938 server_base.cc:1047] running on GCE node
W20250905 02:20:25.794426 32481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:25.795282 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:25.795460 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:25.795572 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038825795562 us; error 0 us; skew 500 ppm
I20250905 02:20:25.795959 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:25.798043 31938 webserver.cc:480] Webserver started at http://127.31.48.190:42455/ using document root <none> and password file <none>
I20250905 02:20:25.798394 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:25.798547 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:25.798758 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:25.799712 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "d4d77197926e4fde9ce8c7cf1b51a02d"
format_stamp: "Formatted at 2025-09-05 02:20:25 on dist-test-slave-q32q"
I20250905 02:20:25.803575 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.005s
I20250905 02:20:25.806298 32486 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:25.806954 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:20:25.807189 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "d4d77197926e4fde9ce8c7cf1b51a02d"
format_stamp: "Formatted at 2025-09-05 02:20:25 on dist-test-slave-q32q"
I20250905 02:20:25.807439 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:25.820634 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:25.821497 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:25.852942 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:39045
I20250905 02:20:25.853029 32537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:39045 every 8 connection(s)
I20250905 02:20:25.856422 32538 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:25.864949 32538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d: Bootstrap starting.
I20250905 02:20:25.868777 32538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:25.872262 32538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d: No bootstrap required, opened a new log
I20250905 02:20:25.874017 32538 raft_consensus.cc:357] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" member_type: VOTER }
I20250905 02:20:25.874389 32538 raft_consensus.cc:383] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:25.874601 32538 raft_consensus.cc:738] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d77197926e4fde9ce8c7cf1b51a02d, State: Initialized, Role: FOLLOWER
I20250905 02:20:25.875155 32538 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" member_type: VOTER }
I20250905 02:20:25.875600 32538 raft_consensus.cc:397] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:25.875820 32538 raft_consensus.cc:491] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:25.876071 32538 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:25.879873 32538 raft_consensus.cc:513] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" member_type: VOTER }
I20250905 02:20:25.880352 32538 leader_election.cc:304] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d77197926e4fde9ce8c7cf1b51a02d; no voters:
I20250905 02:20:25.881397 32538 leader_election.cc:290] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:25.881690 32541 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:25.882938 32541 raft_consensus.cc:695] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 1 LEADER]: Becoming Leader. State: Replica: d4d77197926e4fde9ce8c7cf1b51a02d, State: Running, Role: LEADER
I20250905 02:20:25.883560 32541 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" member_type: VOTER }
I20250905 02:20:25.884112 32538 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:25.886411 32543 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d77197926e4fde9ce8c7cf1b51a02d. Latest consensus state: current_term: 1 leader_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" member_type: VOTER } }
I20250905 02:20:25.886354 32542 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d77197926e4fde9ce8c7cf1b51a02d" member_type: VOTER } }
I20250905 02:20:25.887063 32543 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:25.887168 32542 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:25.893901 32547 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:25.898764 32547 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:25.899806 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:20:25.906414 32547 catalog_manager.cc:1349] Generated new cluster ID: 330922d355ac4d169af8898186007aed
I20250905 02:20:25.906661 32547 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:25.918551 32547 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:25.919600 32547 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:25.940919 32547 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d: Generated new TSK 0
I20250905 02:20:25.941453 32547 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:25.966162 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:25.971566 32559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:25.972810 32560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:25.974603 31938 server_base.cc:1047] running on GCE node
W20250905 02:20:25.974645 32562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:25.975556 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:25.975752 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:25.975910 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038825975892 us; error 0 us; skew 500 ppm
I20250905 02:20:25.976341 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:25.978395 31938 webserver.cc:480] Webserver started at http://127.31.48.129:37769/ using document root <none> and password file <none>
I20250905 02:20:25.978840 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:25.979019 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:25.979250 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:25.980232 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
format_stamp: "Formatted at 2025-09-05 02:20:25 on dist-test-slave-q32q"
I20250905 02:20:25.984222 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 02:20:25.987005 32567 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:25.987654 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:20:25.987916 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
format_stamp: "Formatted at 2025-09-05 02:20:25 on dist-test-slave-q32q"
I20250905 02:20:25.988166 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:26.001618 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:26.002507 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:26.003850 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:26.006059 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:26.006238 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.006441 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:26.006582 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.040845 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:39891
I20250905 02:20:26.040935 32629 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:39891 every 8 connection(s)
I20250905 02:20:26.045169 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:26.051983 32634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:26.053452 32635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:26.055671 31938 server_base.cc:1047] running on GCE node
W20250905 02:20:26.055996 32637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:26.056988 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 02:20:26.057068 32630 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39045
W20250905 02:20:26.057231 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:26.057439 32630 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:26.057453 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038826057437 us; error 0 us; skew 500 ppm
I20250905 02:20:26.058158 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:26.058209 32630 heartbeater.cc:507] Master 127.31.48.190:39045 requested a full tablet report, sending...
I20250905 02:20:26.059999 32503 ts_manager.cc:194] Registered new tserver with Master: 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891)
I20250905 02:20:26.060773 31938 webserver.cc:480] Webserver started at http://127.31.48.130:39593/ using document root <none> and password file <none>
I20250905 02:20:26.061218 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:26.061380 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:26.061625 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:26.061657 32503 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48770
I20250905 02:20:26.062629 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "d3156c20d9704874a9a53667636ce326"
format_stamp: "Formatted at 2025-09-05 02:20:26 on dist-test-slave-q32q"
I20250905 02:20:26.066365 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 02:20:26.069077 32642 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.069715 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:20:26.069960 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "d3156c20d9704874a9a53667636ce326"
format_stamp: "Formatted at 2025-09-05 02:20:26 on dist-test-slave-q32q"
I20250905 02:20:26.070196 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:26.080724 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:26.081614 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:26.082814 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:26.084851 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:26.085049 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.085263 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:26.085412 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.119876 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:38253
I20250905 02:20:26.119935 32704 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:38253 every 8 connection(s)
I20250905 02:20:26.124487 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:26.130641 32708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:26.131198 32709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:26.133776 32711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:26.134510 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:26.135407 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:26.135656 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:26.135850 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038826135833 us; error 0 us; skew 500 ppm
I20250905 02:20:26.136307 32705 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39045
I20250905 02:20:26.136436 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:26.136656 32705 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:26.137315 32705 heartbeater.cc:507] Master 127.31.48.190:39045 requested a full tablet report, sending...
I20250905 02:20:26.138698 31938 webserver.cc:480] Webserver started at http://127.31.48.131:35715/ using document root <none> and password file <none>
I20250905 02:20:26.139278 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:26.139302 32503 ts_manager.cc:194] Registered new tserver with Master: d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:26.139662 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:26.140014 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:26.141194 32503 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48786
I20250905 02:20:26.141414 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
format_stamp: "Formatted at 2025-09-05 02:20:26 on dist-test-slave-q32q"
I20250905 02:20:26.145121 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20250905 02:20:26.147568 32716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.148240 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250905 02:20:26.148531 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
format_stamp: "Formatted at 2025-09-05 02:20:26 on dist-test-slave-q32q"
I20250905 02:20:26.148795 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:26.177692 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:26.178615 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:26.179903 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:26.181892 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:26.182055 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.182287 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:26.182425 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:26.218966 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:41609
I20250905 02:20:26.219053 310 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:41609 every 8 connection(s)
I20250905 02:20:26.230676 311 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39045
I20250905 02:20:26.231007 311 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:26.231704 311 heartbeater.cc:507] Master 127.31.48.190:39045 requested a full tablet report, sending...
I20250905 02:20:26.233387 32503 ts_manager.cc:194] Registered new tserver with Master: 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:26.233901 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.011952353s
I20250905 02:20:26.234634 32503 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48800
I20250905 02:20:27.063868 32630 heartbeater.cc:499] Master 127.31.48.190:39045 was elected leader, sending a full tablet report...
I20250905 02:20:27.143383 32705 heartbeater.cc:499] Master 127.31.48.190:39045 was elected leader, sending a full tablet report...
I20250905 02:20:27.237177 311 heartbeater.cc:499] Master 127.31.48.190:39045 was elected leader, sending a full tablet report...
I20250905 02:20:27.265537 31938 test_util.cc:276] Using random seed: 2066210283
I20250905 02:20:27.286003 32503 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41632:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o""\004\001\000\377\377\377\017\004\001\000\376\377\377\037\004\001\000\375\377\377/\004\001\000\374\377\377?\004\001\000\373\377\377O\004\001\000\372\377\377_\004\001\000\371\377\377o"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:20:27.288502 32503 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:20:27.355847 32670 tablet_service.cc:1468] Processing CreateTablet for tablet d5cfb0a9668b47a7ab7437e0dabae0ee (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250905 02:20:27.357205 32670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cfb0a9668b47a7ab7437e0dabae0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.357437 32668 tablet_service.cc:1468] Processing CreateTablet for tablet 8958e6c9050d48a6af6fb9b333f7282d (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250905 02:20:27.358677 32668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8958e6c9050d48a6af6fb9b333f7282d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.358722 32666 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf808053b644930abd0e5fc8344cd2f (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250905 02:20:27.360837 32667 tablet_service.cc:1468] Processing CreateTablet for tablet 81675d685bdc4b149056a127137aa2d1 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250905 02:20:27.364686 32669 tablet_service.cc:1468] Processing CreateTablet for tablet 4f3dbbe24597498ab2a205e6cdf8d806 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250905 02:20:27.366358 32666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf808053b644930abd0e5fc8344cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.369123 32667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81675d685bdc4b149056a127137aa2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.368888 32744 tablet_service.cc:1468] Processing CreateTablet for tablet d5cfb0a9668b47a7ab7437e0dabae0ee (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250905 02:20:27.370141 32744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cfb0a9668b47a7ab7437e0dabae0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.373602 32669 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3dbbe24597498ab2a205e6cdf8d806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.382539 32743 tablet_service.cc:1468] Processing CreateTablet for tablet 4f3dbbe24597498ab2a205e6cdf8d806 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250905 02:20:27.385656 32743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3dbbe24597498ab2a205e6cdf8d806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.393873 32742 tablet_service.cc:1468] Processing CreateTablet for tablet 8958e6c9050d48a6af6fb9b333f7282d (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250905 02:20:27.395298 32742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8958e6c9050d48a6af6fb9b333f7282d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.396919 32595 tablet_service.cc:1468] Processing CreateTablet for tablet d5cfb0a9668b47a7ab7437e0dabae0ee (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION VALUES < 268435455
I20250905 02:20:27.398255 32595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5cfb0a9668b47a7ab7437e0dabae0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.400049 32739 tablet_service.cc:1468] Processing CreateTablet for tablet 61191ceca6324c52a4c2e7669736a8c1 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250905 02:20:27.401401 32739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61191ceca6324c52a4c2e7669736a8c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.401177 32740 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf808053b644930abd0e5fc8344cd2f (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250905 02:20:27.401597 32594 tablet_service.cc:1468] Processing CreateTablet for tablet 4f3dbbe24597498ab2a205e6cdf8d806 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 268435455 <= VALUES < 536870910
I20250905 02:20:27.402426 32740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf808053b644930abd0e5fc8344cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.400049 32741 tablet_service.cc:1468] Processing CreateTablet for tablet 81675d685bdc4b149056a127137aa2d1 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250905 02:20:27.402956 32594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f3dbbe24597498ab2a205e6cdf8d806. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.403723 32741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81675d685bdc4b149056a127137aa2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.407640 32592 tablet_service.cc:1468] Processing CreateTablet for tablet 81675d685bdc4b149056a127137aa2d1 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 805306365 <= VALUES < 1073741820
I20250905 02:20:27.409534 32591 tablet_service.cc:1468] Processing CreateTablet for tablet 2cf808053b644930abd0e5fc8344cd2f (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1342177275
I20250905 02:20:27.410941 32591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cf808053b644930abd0e5fc8344cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.410581 32590 tablet_service.cc:1468] Processing CreateTablet for tablet 61191ceca6324c52a4c2e7669736a8c1 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250905 02:20:27.411909 32590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61191ceca6324c52a4c2e7669736a8c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.414673 32666 tablet_service.cc:1468] Processing CreateTablet for tablet e5aba5dd131a405885e8666e93aed51d (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250905 02:20:27.415009 32665 tablet_service.cc:1468] Processing CreateTablet for tablet 7155cc6ee3994fc88cf33b4715eb443a (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250905 02:20:27.415962 32666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5aba5dd131a405885e8666e93aed51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.406246 32593 tablet_service.cc:1468] Processing CreateTablet for tablet 8958e6c9050d48a6af6fb9b333f7282d (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 536870910 <= VALUES < 805306365
I20250905 02:20:27.418593 32593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8958e6c9050d48a6af6fb9b333f7282d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.421147 32592 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81675d685bdc4b149056a127137aa2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.423511 32665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7155cc6ee3994fc88cf33b4715eb443a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.433445 32667 tablet_service.cc:1468] Processing CreateTablet for tablet 61191ceca6324c52a4c2e7669736a8c1 (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1342177275 <= VALUES < 1610612730
I20250905 02:20:27.427651 32589 tablet_service.cc:1468] Processing CreateTablet for tablet e5aba5dd131a405885e8666e93aed51d (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250905 02:20:27.435076 32667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61191ceca6324c52a4c2e7669736a8c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.441269 331 tablet_bootstrap.cc:492] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.444669 32737 tablet_service.cc:1468] Processing CreateTablet for tablet 7155cc6ee3994fc88cf33b4715eb443a (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250905 02:20:27.445960 32737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7155cc6ee3994fc88cf33b4715eb443a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.451746 32589 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5aba5dd131a405885e8666e93aed51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.430616 32588 tablet_service.cc:1468] Processing CreateTablet for tablet 7155cc6ee3994fc88cf33b4715eb443a (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1879048185 <= VALUES
I20250905 02:20:27.453725 32588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7155cc6ee3994fc88cf33b4715eb443a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.470592 32738 tablet_service.cc:1468] Processing CreateTablet for tablet e5aba5dd131a405885e8666e93aed51d (DEFAULT_TABLE table=test-workload [id=5a60b478be6e4909bd94dd900c527527]), partition=RANGE (key) PARTITION 1610612730 <= VALUES < 1879048185
I20250905 02:20:27.478534 32738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5aba5dd131a405885e8666e93aed51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:27.494045 332 tablet_bootstrap.cc:492] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.516933 331 tablet_bootstrap.cc:654] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.522656 332 tablet_bootstrap.cc:654] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.525094 333 tablet_bootstrap.cc:492] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.542088 332 tablet_bootstrap.cc:492] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:27.542622 332 ts_tablet_manager.cc:1397] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.004s
I20250905 02:20:27.546983 333 tablet_bootstrap.cc:654] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.551195 331 tablet_bootstrap.cc:492] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.551589 331 ts_tablet_manager.cc:1397] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.004s
I20250905 02:20:27.553859 331 raft_consensus.cc:357] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.554567 331 raft_consensus.cc:383] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.554277 332 raft_consensus.cc:357] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.554865 331 raft_consensus.cc:738] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.554944 332 raft_consensus.cc:383] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.555363 332 raft_consensus.cc:738] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.555610 331 consensus_queue.cc:260] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.556046 332 consensus_queue.cc:260] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.558650 331 ts_tablet_manager.cc:1428] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250905 02:20:27.559072 333 tablet_bootstrap.cc:492] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.559522 331 tablet_bootstrap.cc:492] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.559667 333 ts_tablet_manager.cc:1397] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.000s
I20250905 02:20:27.563028 333 raft_consensus.cc:357] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.563966 333 raft_consensus.cc:383] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.564313 333 raft_consensus.cc:738] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.565645 331 tablet_bootstrap.cc:654] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.565313 333 consensus_queue.cc:260] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.573499 332 ts_tablet_manager.cc:1428] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.031s user 0.012s sys 0.010s
I20250905 02:20:27.574487 332 tablet_bootstrap.cc:492] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.575434 333 ts_tablet_manager.cc:1428] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20250905 02:20:27.579268 333 tablet_bootstrap.cc:492] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.581146 332 tablet_bootstrap.cc:654] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.584717 333 tablet_bootstrap.cc:654] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.589332 331 tablet_bootstrap.cc:492] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.589144 337 raft_consensus.cc:491] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.589744 331 ts_tablet_manager.cc:1397] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.012s
I20250905 02:20:27.589864 337 raft_consensus.cc:513] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.591799 331 raft_consensus.cc:357] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.592386 331 raft_consensus.cc:383] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.592679 331 raft_consensus.cc:738] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.593402 331 consensus_queue.cc:260] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.595582 337 leader_election.cc:290] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609), 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891)
I20250905 02:20:27.597704 332 tablet_bootstrap.cc:492] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:27.598177 332 ts_tablet_manager.cc:1397] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.007s
I20250905 02:20:27.601492 332 raft_consensus.cc:357] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.602358 332 raft_consensus.cc:383] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.602648 332 raft_consensus.cc:738] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.603734 333 tablet_bootstrap.cc:492] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.603420 332 consensus_queue.cc:260] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.604185 333 ts_tablet_manager.cc:1397] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.004s
I20250905 02:20:27.605898 332 ts_tablet_manager.cc:1428] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 02:20:27.606988 332 tablet_bootstrap.cc:492] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.606698 333 raft_consensus.cc:357] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.607403 333 raft_consensus.cc:383] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.607745 333 raft_consensus.cc:738] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.613943 332 tablet_bootstrap.cc:654] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.619400 331 ts_tablet_manager.cc:1428] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.029s user 0.009s sys 0.006s
I20250905 02:20:27.620468 331 tablet_bootstrap.cc:492] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.620702 333 consensus_queue.cc:260] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.626366 332 tablet_bootstrap.cc:492] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:27.626819 332 ts_tablet_manager.cc:1397] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.008s
I20250905 02:20:27.627415 331 tablet_bootstrap.cc:654] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.627995 333 ts_tablet_manager.cc:1428] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250905 02:20:27.630332 333 tablet_bootstrap.cc:492] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.629407 332 raft_consensus.cc:357] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.635373 332 raft_consensus.cc:383] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.636006 332 raft_consensus.cc:738] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.636183 32754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cfb0a9668b47a7ab7437e0dabae0ee" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:27.636868 332 consensus_queue.cc:260] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.638137 331 tablet_bootstrap.cc:492] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.638574 331 ts_tablet_manager.cc:1397] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.003s
I20250905 02:20:27.639658 332 ts_tablet_manager.cc:1428] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250905 02:20:27.640573 332 tablet_bootstrap.cc:492] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.641589 331 raft_consensus.cc:357] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.642328 331 raft_consensus.cc:383] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.642623 331 raft_consensus.cc:738] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
W20250905 02:20:27.643430 32643 leader_election.cc:343] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.643270 331 consensus_queue.cc:260] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.647018 332 tablet_bootstrap.cc:654] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.649415 32605 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cfb0a9668b47a7ab7437e0dabae0ee" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:27.649755 333 tablet_bootstrap.cc:654] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.650267 331 ts_tablet_manager.cc:1428] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.011s user 0.003s sys 0.006s
I20250905 02:20:27.651305 331 tablet_bootstrap.cc:492] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
W20250905 02:20:27.652806 32646 leader_election.cc:343] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.653429 32646 leader_election.cc:304] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3156c20d9704874a9a53667636ce326; no voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa
I20250905 02:20:27.654965 345 raft_consensus.cc:2747] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 02:20:27.658854 331 tablet_bootstrap.cc:654] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.662629 333 tablet_bootstrap.cc:492] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.663515 333 ts_tablet_manager.cc:1397] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.008s
I20250905 02:20:27.668568 333 raft_consensus.cc:357] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.669719 333 raft_consensus.cc:383] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.670183 333 raft_consensus.cc:738] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.671455 333 consensus_queue.cc:260] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.678509 333 ts_tablet_manager.cc:1428] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20250905 02:20:27.680260 333 tablet_bootstrap.cc:492] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.683646 331 tablet_bootstrap.cc:492] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.684139 331 ts_tablet_manager.cc:1397] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.023s
I20250905 02:20:27.686663 331 raft_consensus.cc:357] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.687378 331 raft_consensus.cc:383] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.687709 331 raft_consensus.cc:738] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.688567 333 tablet_bootstrap.cc:654] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.688436 331 consensus_queue.cc:260] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.691124 331 ts_tablet_manager.cc:1428] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 02:20:27.692302 331 tablet_bootstrap.cc:492] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.699659 331 tablet_bootstrap.cc:654] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.701350 333 tablet_bootstrap.cc:492] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.701931 333 ts_tablet_manager.cc:1397] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.008s
I20250905 02:20:27.705363 333 raft_consensus.cc:357] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.706591 333 raft_consensus.cc:383] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.707047 333 raft_consensus.cc:738] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.708779 333 consensus_queue.cc:260] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.710573 332 tablet_bootstrap.cc:492] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:27.710851 331 tablet_bootstrap.cc:492] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.711088 332 ts_tablet_manager.cc:1397] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.071s user 0.047s sys 0.018s
I20250905 02:20:27.711246 333 ts_tablet_manager.cc:1428] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250905 02:20:27.711267 331 ts_tablet_manager.cc:1397] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20250905 02:20:27.712453 333 tablet_bootstrap.cc:492] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.714159 331 raft_consensus.cc:357] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.714358 332 raft_consensus.cc:357] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.714874 331 raft_consensus.cc:383] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.714980 332 raft_consensus.cc:383] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.715287 332 raft_consensus.cc:738] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.715212 331 raft_consensus.cc:738] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.716427 331 consensus_queue.cc:260] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.716437 332 consensus_queue.cc:260] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.719058 331 ts_tablet_manager.cc:1428] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 02:20:27.719182 333 tablet_bootstrap.cc:654] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.720147 331 tablet_bootstrap.cc:492] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.726835 331 tablet_bootstrap.cc:654] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.726938 332 ts_tablet_manager.cc:1428] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.016s user 0.000s sys 0.008s
I20250905 02:20:27.730898 332 tablet_bootstrap.cc:492] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.738329 333 tablet_bootstrap.cc:492] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.738801 333 ts_tablet_manager.cc:1397] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.011s
I20250905 02:20:27.738919 332 tablet_bootstrap.cc:654] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.742514 333 raft_consensus.cc:357] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.743076 333 raft_consensus.cc:383] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.743367 333 raft_consensus.cc:738] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.744025 333 consensus_queue.cc:260] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.744530 332 tablet_bootstrap.cc:492] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:27.744709 331 tablet_bootstrap.cc:492] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.745122 331 ts_tablet_manager.cc:1397] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.008s
I20250905 02:20:27.745340 332 ts_tablet_manager.cc:1397] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.012s
I20250905 02:20:27.746775 333 ts_tablet_manager.cc:1428] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20250905 02:20:27.747699 333 tablet_bootstrap.cc:492] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.747488 331 raft_consensus.cc:357] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.747910 332 raft_consensus.cc:357] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.748590 332 raft_consensus.cc:383] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.748754 331 raft_consensus.cc:383] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.748869 332 raft_consensus.cc:738] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.749174 331 raft_consensus.cc:738] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.749729 332 consensus_queue.cc:260] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.749974 331 consensus_queue.cc:260] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.752208 331 ts_tablet_manager.cc:1428] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20250905 02:20:27.753177 331 tablet_bootstrap.cc:492] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.754904 333 tablet_bootstrap.cc:654] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.755844 339 raft_consensus.cc:491] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.756297 339 raft_consensus.cc:513] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.759374 339 leader_election.cc:290] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:27.767040 338 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.767720 338 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.771620 345 raft_consensus.cc:491] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.771054 338 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.772133 345 raft_consensus.cc:513] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.777046 332 ts_tablet_manager.cc:1428] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.031s user 0.007s sys 0.000s
I20250905 02:20:27.777876 345 leader_election.cc:290] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.778285 332 tablet_bootstrap.cc:492] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.788851 32605 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7155cc6ee3994fc88cf33b4715eb443a" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:27.789235 333 tablet_bootstrap.cc:492] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.789677 333 ts_tablet_manager.cc:1397] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.042s user 0.016s sys 0.012s
I20250905 02:20:27.789507 32605 raft_consensus.cc:2466] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 0.
I20250905 02:20:27.785688 32754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7155cc6ee3994fc88cf33b4715eb443a" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:27.791455 32646 leader_election.cc:304] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:27.792397 345 raft_consensus.cc:2802] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.792754 345 raft_consensus.cc:491] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 02:20:27.793030 32643 leader_election.cc:343] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.793340 332 tablet_bootstrap.cc:654] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.786983 331 tablet_bootstrap.cc:654] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.793041 345 raft_consensus.cc:3058] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.793526 333 raft_consensus.cc:357] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.806443 333 raft_consensus.cc:383] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.809047 333 raft_consensus.cc:738] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.799520 339 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.813244 339 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.812996 333 consensus_queue.cc:260] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.815697 345 raft_consensus.cc:513] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.819653 32605 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7155cc6ee3994fc88cf33b4715eb443a" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
I20250905 02:20:27.820200 32605 raft_consensus.cc:3058] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.818651 345 leader_election.cc:290] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 election: Requested vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.823658 32754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7155cc6ee3994fc88cf33b4715eb443a" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
I20250905 02:20:27.825138 32754 raft_consensus.cc:3058] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.823361 339 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:27.827877 338 raft_consensus.cc:491] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.830679 338 raft_consensus.cc:513] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.831947 333 ts_tablet_manager.cc:1428] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.041s user 0.008s sys 0.005s
I20250905 02:20:27.832906 333 tablet_bootstrap.cc:492] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.833333 338 leader_election.cc:290] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.834980 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:27.835637 32753 raft_consensus.cc:2466] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 0.
I20250905 02:20:27.842341 331 tablet_bootstrap.cc:492] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.842857 331 ts_tablet_manager.cc:1397] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.006s
I20250905 02:20:27.845255 32754 raft_consensus.cc:2466] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 1.
I20250905 02:20:27.845652 331 raft_consensus.cc:357] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.846419 331 raft_consensus.cc:383] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.846714 331 raft_consensus.cc:738] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.847361 331 consensus_queue.cc:260] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.848649 32680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8958e6c9050d48a6af6fb9b333f7282d" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.850364 32679 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.851329 32680 raft_consensus.cc:2466] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 0.
I20250905 02:20:27.851507 32643 leader_election.cc:304] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:27.853830 345 raft_consensus.cc:2802] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:27.854671 345 raft_consensus.cc:695] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 1 LEADER]: Becoming Leader. State: Replica: d3156c20d9704874a9a53667636ce326, State: Running, Role: LEADER
I20250905 02:20:27.852669 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf808053b644930abd0e5fc8344cd2f" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:27.855605 32753 raft_consensus.cc:2466] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 0.
I20250905 02:20:27.855542 345 consensus_queue.cc:237] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
W20250905 02:20:27.854664 32719 leader_election.cc:343] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.859107 32719 leader_election.cc:304] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:27.859333 32605 raft_consensus.cc:2466] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 1.
I20250905 02:20:27.861649 339 raft_consensus.cc:2802] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.862170 339 raft_consensus.cc:491] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:27.863140 339 raft_consensus.cc:3058] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.866961 332 tablet_bootstrap.cc:492] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:27.867745 332 ts_tablet_manager.cc:1397] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.021s
I20250905 02:20:27.870847 339 raft_consensus.cc:513] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.872056 332 raft_consensus.cc:357] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.875562 332 raft_consensus.cc:383] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.875227 331 ts_tablet_manager.cc:1428] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20250905 02:20:27.874600 333 tablet_bootstrap.cc:654] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.876349 332 raft_consensus.cc:738] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.876479 32604 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:27.876605 32605 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8958e6c9050d48a6af6fb9b333f7282d" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:27.880882 32604 raft_consensus.cc:2466] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 0.
I20250905 02:20:27.861915 32568 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa; no voters:
I20250905 02:20:27.882622 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8958e6c9050d48a6af6fb9b333f7282d" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
W20250905 02:20:27.882164 32720 leader_election.cc:343] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.883772 32680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8958e6c9050d48a6af6fb9b333f7282d" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:27.884095 32720 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa; no voters: d3156c20d9704874a9a53667636ce326
I20250905 02:20:27.884243 338 raft_consensus.cc:2802] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.884723 338 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:27.885066 338 raft_consensus.cc:3058] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.883014 339 leader_election.cc:290] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Requested vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:27.885744 331 tablet_bootstrap.cc:492] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326: Bootstrap starting.
I20250905 02:20:27.886050 362 raft_consensus.cc:2802] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.884348 32680 raft_consensus.cc:3058] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
W20250905 02:20:27.887787 32568 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.31.48.131:41609, user_credentials={real_user=slave}} blocked reactor thread for 44232.6us
I20250905 02:20:27.889431 362 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:27.890023 362 raft_consensus.cc:3058] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.880319 332 consensus_queue.cc:260] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
W20250905 02:20:27.885275 32720 leader_election.cc:343] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891): Illegal state: must be running to vote when last-logged opid is not known
W20250905 02:20:27.899216 32551 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 02:20:27.901651 32556 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20250905 02:20:27.902992 32556 auto_leader_rebalancer.cc:460] the master instance isn't leader: Service unavailable: Tablet not running
I20250905 02:20:27.896469 32501 catalog_manager.cc:5582] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3156c20d9704874a9a53667636ce326 (127.31.48.130). New cstate: current_term: 1 leader_uuid: "d3156c20d9704874a9a53667636ce326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:27.905769 32679 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.907476 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf808053b644930abd0e5fc8344cd2f" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.898662 32568 leader_election.cc:304] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa; no voters:
I20250905 02:20:27.908192 32678 raft_consensus.cc:2466] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 0.
W20250905 02:20:27.908735 32570 leader_election.cc:343] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.908299 362 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.918145 331 tablet_bootstrap.cc:654] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.919942 362 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Requested vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:27.910703 333 tablet_bootstrap.cc:492] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:27.921428 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:27.921797 338 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.922384 333 ts_tablet_manager.cc:1397] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.090s user 0.019s sys 0.004s
I20250905 02:20:27.922422 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
W20250905 02:20:27.923050 32719 leader_election.cc:343] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.923511 332 ts_tablet_manager.cc:1428] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.055s user 0.015s sys 0.000s
I20250905 02:20:27.923815 32603 raft_consensus.cc:2391] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4f5319a94bc54262a8ce9b7e5fea47fa in current term 1: Already voted for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in this term.
I20250905 02:20:27.924955 338 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Requested vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.925179 367 raft_consensus.cc:2802] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.925457 332 tablet_bootstrap.cc:492] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:27.925850 367 raft_consensus.cc:491] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:27.925988 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:27.926297 367 raft_consensus.cc:3058] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.927785 32720 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f5319a94bc54262a8ce9b7e5fea47fa; no voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326
W20250905 02:20:27.929091 32570 leader_election.cc:343] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:20:27.929322 362 raft_consensus.cc:2747] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 02:20:27.929893 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
I20250905 02:20:27.930913 32753 raft_consensus.cc:2391] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4ba0c6edbed6407bbb45011d4adc4d61 in current term 1: Already voted for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in this term.
I20250905 02:20:27.930059 333 raft_consensus.cc:357] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.931699 333 raft_consensus.cc:383] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.931998 333 raft_consensus.cc:738] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.932188 32568 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61; no voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326
I20250905 02:20:27.932909 32680 raft_consensus.cc:2466] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 1.
I20250905 02:20:27.932931 333 consensus_queue.cc:260] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.933538 338 raft_consensus.cc:2747] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 02:20:27.934787 32719 leader_election.cc:304] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326; no voters: 4ba0c6edbed6407bbb45011d4adc4d61
I20250905 02:20:27.936631 333 ts_tablet_manager.cc:1428] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250905 02:20:27.937759 362 raft_consensus.cc:2802] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:27.938414 362 raft_consensus.cc:695] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Becoming Leader. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Running, Role: LEADER
I20250905 02:20:27.938539 333 tablet_bootstrap.cc:492] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa: Bootstrap starting.
I20250905 02:20:27.940330 362 consensus_queue.cc:237] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.944446 339 raft_consensus.cc:491] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.944949 339 raft_consensus.cc:513] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:27.947065 339 leader_election.cc:290] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891)
I20250905 02:20:27.947546 339 raft_consensus.cc:491] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.953889 332 tablet_bootstrap.cc:654] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.953756 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cfb0a9668b47a7ab7437e0dabae0ee" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:27.947971 339 raft_consensus.cc:513] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.954736 32603 raft_consensus.cc:2466] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 0.
I20250905 02:20:27.956434 367 raft_consensus.cc:513] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.957319 32680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cfb0a9668b47a7ab7437e0dabae0ee" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.957441 32720 leader_election.cc:304] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa; no voters:
I20250905 02:20:27.958613 32680 raft_consensus.cc:2466] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 0.
I20250905 02:20:27.959976 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81675d685bdc4b149056a127137aa2d1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:27.960577 32603 raft_consensus.cc:2466] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 0.
I20250905 02:20:27.960834 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf808053b644930abd0e5fc8344cd2f" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:27.961910 32678 raft_consensus.cc:3058] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.966178 331 tablet_bootstrap.cc:492] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326: No bootstrap required, opened a new log
I20250905 02:20:27.966624 331 ts_tablet_manager.cc:1397] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.009s
I20250905 02:20:27.966404 32720 leader_election.cc:304] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa; no voters:
I20250905 02:20:27.967938 339 leader_election.cc:290] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:27.969723 362 raft_consensus.cc:2802] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.970089 362 raft_consensus.cc:491] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:27.970427 362 raft_consensus.cc:3058] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.965296 367 leader_election.cc:290] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Requested vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.972350 338 raft_consensus.cc:491] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:27.972962 338 raft_consensus.cc:513] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.968701 333 tablet_bootstrap.cc:654] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:27.972067 331 raft_consensus.cc:357] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.977931 331 raft_consensus.cc:383] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:27.978330 331 raft_consensus.cc:738] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3156c20d9704874a9a53667636ce326, State: Initialized, Role: FOLLOWER
I20250905 02:20:27.977432 32679 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f3dbbe24597498ab2a205e6cdf8d806" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.979928 32679 raft_consensus.cc:2466] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 0.
I20250905 02:20:27.981509 338 leader_election.cc:290] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:27.984151 331 consensus_queue.cc:260] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:27.983762 32680 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81675d685bdc4b149056a127137aa2d1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:27.985141 32680 raft_consensus.cc:2466] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 0.
I20250905 02:20:27.986002 32570 leader_election.cc:304] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:27.986815 331 ts_tablet_manager.cc:1428] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326: Time spent starting tablet: real 0.020s user 0.006s sys 0.003s
I20250905 02:20:27.987376 338 raft_consensus.cc:2802] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:27.987905 338 raft_consensus.cc:491] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:27.988386 338 raft_consensus.cc:3058] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:27.982334 32502 catalog_manager.cc:5582] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131). New cstate: current_term: 1 leader_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:27.997519 32678 raft_consensus.cc:2466] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 1.
I20250905 02:20:27.997577 362 raft_consensus.cc:513] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.004568 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2cf808053b644930abd0e5fc8344cd2f" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
I20250905 02:20:28.005235 32753 raft_consensus.cc:3058] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.006099 362 leader_election.cc:290] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Requested vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253)
I20250905 02:20:28.001808 32570 leader_election.cc:304] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:28.011386 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81675d685bdc4b149056a127137aa2d1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:28.011956 371 raft_consensus.cc:2802] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:28.012879 371 raft_consensus.cc:491] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:28.013211 371 raft_consensus.cc:3058] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.013594 338 raft_consensus.cc:513] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.012168 32678 raft_consensus.cc:3058] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.015208 32754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f3dbbe24597498ab2a205e6cdf8d806" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:28.015823 32754 raft_consensus.cc:2466] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 0.
I20250905 02:20:28.013942 367 raft_consensus.cc:2802] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:28.022035 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "81675d685bdc4b149056a127137aa2d1" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
I20250905 02:20:28.022739 32603 raft_consensus.cc:3058] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.026648 32753 raft_consensus.cc:2466] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 1.
I20250905 02:20:28.027359 32678 raft_consensus.cc:2466] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 1.
I20250905 02:20:28.028419 32719 leader_election.cc:304] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:28.029281 362 raft_consensus.cc:2802] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:28.029757 362 raft_consensus.cc:695] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Becoming Leader. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Running, Role: LEADER
I20250905 02:20:28.030342 332 tablet_bootstrap.cc:492] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:28.030941 332 ts_tablet_manager.cc:1397] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.106s user 0.004s sys 0.023s
I20250905 02:20:28.030501 362 consensus_queue.cc:237] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.035759 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f3dbbe24597498ab2a205e6cdf8d806" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:28.036715 32678 raft_consensus.cc:3058] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.038942 32603 raft_consensus.cc:2466] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 1.
I20250905 02:20:28.038494 332 raft_consensus.cc:357] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.040591 332 raft_consensus.cc:383] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:28.041414 338 leader_election.cc:290] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Requested vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:28.041476 332 raft_consensus.cc:738] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:28.043406 32678 raft_consensus.cc:2466] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 1.
I20250905 02:20:28.044512 32570 leader_election.cc:304] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:28.044839 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f3dbbe24597498ab2a205e6cdf8d806" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
I20250905 02:20:28.045414 338 raft_consensus.cc:2802] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:28.045596 32753 raft_consensus.cc:3058] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.046036 338 raft_consensus.cc:695] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 LEADER]: Becoming Leader. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Running, Role: LEADER
I20250905 02:20:28.047137 338 consensus_queue.cc:237] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.048718 332 consensus_queue.cc:260] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.053215 371 raft_consensus.cc:513] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:28.056891 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cfb0a9668b47a7ab7437e0dabae0ee" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:28.057466 32678 raft_consensus.cc:3058] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.059092 367 raft_consensus.cc:695] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 LEADER]: Becoming Leader. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Running, Role: LEADER
I20250905 02:20:28.059949 367 consensus_queue.cc:237] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.066612 371 leader_election.cc:290] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Requested vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891)
I20250905 02:20:28.067982 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d5cfb0a9668b47a7ab7437e0dabae0ee" candidate_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
I20250905 02:20:28.068596 32603 raft_consensus.cc:3058] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.069108 32678 raft_consensus.cc:2466] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 1.
I20250905 02:20:28.070242 32719 leader_election.cc:304] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:28.070458 32753 raft_consensus.cc:2466] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 1.
I20250905 02:20:28.071606 333 tablet_bootstrap.cc:492] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa: No bootstrap required, opened a new log
I20250905 02:20:28.071655 332 ts_tablet_manager.cc:1428] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.040s user 0.009s sys 0.003s
I20250905 02:20:28.072242 333 ts_tablet_manager.cc:1397] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent bootstrapping tablet: real 0.134s user 0.016s sys 0.014s
I20250905 02:20:28.072881 332 tablet_bootstrap.cc:492] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61: Bootstrap starting.
I20250905 02:20:28.074833 333 raft_consensus.cc:357] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.075488 333 raft_consensus.cc:383] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:28.075766 333 raft_consensus.cc:738] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Initialized, Role: FOLLOWER
I20250905 02:20:28.076680 371 raft_consensus.cc:2802] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:28.077250 371 raft_consensus.cc:695] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Becoming Leader. State: Replica: 4f5319a94bc54262a8ce9b7e5fea47fa, State: Running, Role: LEADER
I20250905 02:20:28.077737 32603 raft_consensus.cc:2466] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in term 1.
I20250905 02:20:28.078080 371 consensus_queue.cc:237] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } }
I20250905 02:20:28.083096 333 consensus_queue.cc:260] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.085549 333 ts_tablet_manager.cc:1428] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250905 02:20:28.090514 32501 catalog_manager.cc:5582] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131). New cstate: current_term: 1 leader_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:28.090399 32502 catalog_manager.cc:5582] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:28.095760 332 tablet_bootstrap.cc:654] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:28.102277 332 tablet_bootstrap.cc:492] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61: No bootstrap required, opened a new log
I20250905 02:20:28.102641 332 ts_tablet_manager.cc:1397] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.011s
I20250905 02:20:28.104678 332 raft_consensus.cc:357] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.105160 332 raft_consensus.cc:383] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:28.105362 332 raft_consensus.cc:738] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Initialized, Role: FOLLOWER
I20250905 02:20:28.105868 332 consensus_queue.cc:260] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.106379 32502 catalog_manager.cc:5582] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131). New cstate: current_term: 1 leader_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:28.107288 332 ts_tablet_manager.cc:1428] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20250905 02:20:28.109675 32501 catalog_manager.cc:5582] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:28.122953 345 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:28.123517 345 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.125173 345 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:28.125916 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:28.126148 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:28.126705 32603 raft_consensus.cc:2391] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3156c20d9704874a9a53667636ce326 in current term 1: Already voted for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in this term.
I20250905 02:20:28.126888 32753 raft_consensus.cc:2391] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3156c20d9704874a9a53667636ce326 in current term 1: Already voted for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in this term.
I20250905 02:20:28.128149 32646 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3156c20d9704874a9a53667636ce326; no voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa
I20250905 02:20:28.128801 345 raft_consensus.cc:3058] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.133384 345 raft_consensus.cc:2747] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 02:20:28.253376 345 raft_consensus.cc:491] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:28.253850 345 raft_consensus.cc:513] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.255342 345 leader_election.cc:290] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:28.256232 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5aba5dd131a405885e8666e93aed51d" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" is_pre_election: true
I20250905 02:20:28.256345 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5aba5dd131a405885e8666e93aed51d" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:28.256775 32603 raft_consensus.cc:2466] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 0.
I20250905 02:20:28.256937 32753 raft_consensus.cc:2466] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 0.
I20250905 02:20:28.257628 32646 leader_election.cc:304] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:28.258196 345 raft_consensus.cc:2802] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:28.258551 345 raft_consensus.cc:491] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:28.258818 345 raft_consensus.cc:3058] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.263029 345 raft_consensus.cc:513] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.264523 345 leader_election.cc:290] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 election: Requested vote from peers 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:28.265265 32603 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5aba5dd131a405885e8666e93aed51d" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ba0c6edbed6407bbb45011d4adc4d61"
I20250905 02:20:28.265393 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5aba5dd131a405885e8666e93aed51d" candidate_uuid: "d3156c20d9704874a9a53667636ce326" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
I20250905 02:20:28.265729 32603 raft_consensus.cc:3058] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.265869 32753 raft_consensus.cc:3058] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.270093 32603 raft_consensus.cc:2466] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 1.
I20250905 02:20:28.270195 32753 raft_consensus.cc:2466] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3156c20d9704874a9a53667636ce326 in term 1.
I20250905 02:20:28.271039 32646 leader_election.cc:304] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:28.271720 345 raft_consensus.cc:2802] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:28.272200 345 raft_consensus.cc:695] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 1 LEADER]: Becoming Leader. State: Replica: d3156c20d9704874a9a53667636ce326, State: Running, Role: LEADER
I20250905 02:20:28.272868 345 consensus_queue.cc:237] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.279666 32501 catalog_manager.cc:5582] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3156c20d9704874a9a53667636ce326 (127.31.48.130). New cstate: current_term: 1 leader_uuid: "d3156c20d9704874a9a53667636ce326" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:28.365090 367 raft_consensus.cc:491] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:28.365504 367 raft_consensus.cc:513] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:28.367007 367 leader_election.cc:290] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:28.367853 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8958e6c9050d48a6af6fb9b333f7282d" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:28.367951 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8958e6c9050d48a6af6fb9b333f7282d" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:28.368484 32678 raft_consensus.cc:2391] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4ba0c6edbed6407bbb45011d4adc4d61 in current term 1: Already voted for candidate 4f5319a94bc54262a8ce9b7e5fea47fa in this term.
I20250905 02:20:28.369248 32570 leader_election.cc:304] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61; no voters: 4f5319a94bc54262a8ce9b7e5fea47fa, d3156c20d9704874a9a53667636ce326
I20250905 02:20:28.369781 367 raft_consensus.cc:3058] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:28.374039 367 raft_consensus.cc:2747] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 02:20:28.423348 345 consensus_queue.cc:1035] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.433794 345 consensus_queue.cc:1035] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.479219 367 consensus_queue.cc:1035] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.490132 338 consensus_queue.cc:1035] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.493423 371 consensus_queue.cc:1035] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.507990 373 consensus_queue.cc:1035] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:20:28.517620 373 consensus_queue.cc:1035] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.529484 373 consensus_queue.cc:1035] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.553597 373 consensus_queue.cc:1035] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.560123 338 consensus_queue.cc:1035] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.565930 371 consensus_queue.cc:1035] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.583371 367 consensus_queue.cc:1035] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.675696 337 consensus_queue.cc:1035] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:28.685847 337 consensus_queue.cc:1035] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:20:28.903694 32551 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 02:20:29.509234 403 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:29.509675 403 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:29.511176 403 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:29.512001 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326" is_pre_election: true
I20250905 02:20:29.512184 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" is_pre_election: true
I20250905 02:20:29.512503 32678 raft_consensus.cc:2466] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 1.
I20250905 02:20:29.512742 32753 raft_consensus.cc:2466] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 1.
I20250905 02:20:29.513520 32570 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, d3156c20d9704874a9a53667636ce326; no voters:
I20250905 02:20:29.514132 403 raft_consensus.cc:2802] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:20:29.514397 403 raft_consensus.cc:491] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:29.514634 403 raft_consensus.cc:3058] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:29.518795 403 raft_consensus.cc:513] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:29.520162 403 leader_election.cc:290] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 2 election: Requested vote from peers d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253), 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:29.520977 32678 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3156c20d9704874a9a53667636ce326"
I20250905 02:20:29.521111 32753 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61191ceca6324c52a4c2e7669736a8c1" candidate_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa"
I20250905 02:20:29.521432 32678 raft_consensus.cc:3058] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:29.521560 32753 raft_consensus.cc:3058] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:29.525615 32753 raft_consensus.cc:2466] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 2.
I20250905 02:20:29.525611 32678 raft_consensus.cc:2466] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ba0c6edbed6407bbb45011d4adc4d61 in term 2.
I20250905 02:20:29.526615 32568 leader_election.cc:304] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ba0c6edbed6407bbb45011d4adc4d61, 4f5319a94bc54262a8ce9b7e5fea47fa; no voters:
I20250905 02:20:29.527251 403 raft_consensus.cc:2802] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:20:29.527709 403 raft_consensus.cc:695] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 2 LEADER]: Becoming Leader. State: Replica: 4ba0c6edbed6407bbb45011d4adc4d61, State: Running, Role: LEADER
I20250905 02:20:29.528255 403 consensus_queue.cc:237] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } }
I20250905 02:20:29.535948 32501 catalog_manager.cc:5582] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 reported cstate change: term changed from 0 to 2, leader changed from <none> to 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129). New cstate: current_term: 2 leader_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:29.904083 32556 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:29.911216 32556 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:29.911695 32556 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:30.016054 32678 raft_consensus.cc:1273] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 2 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250905 02:20:30.017143 403 consensus_queue.cc:1035] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:30.030086 32753 raft_consensus.cc:1273] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 2 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250905 02:20:30.031380 412 consensus_queue.cc:1035] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:30.053897 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:30.059806 417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:30.060611 416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:30.061327 419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:30.061789 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:30.062553 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:30.062753 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:30.062888 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038830062877 us; error 0 us; skew 500 ppm
I20250905 02:20:30.063300 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:30.065362 31938 webserver.cc:480] Webserver started at http://127.31.48.132:44411/ using document root <none> and password file <none>
I20250905 02:20:30.065771 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:30.065925 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:30.066128 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:30.067111 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "6e05e6597ea1466e8d73ba147828c300"
format_stamp: "Formatted at 2025-09-05 02:20:30 on dist-test-slave-q32q"
I20250905 02:20:30.071209 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 02:20:30.074087 424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:30.074728 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250905 02:20:30.074994 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "6e05e6597ea1466e8d73ba147828c300"
format_stamp: "Formatted at 2025-09-05 02:20:30 on dist-test-slave-q32q"
I20250905 02:20:30.075239 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:30.099490 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:30.100894 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:30.102586 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:30.104931 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:30.105119 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:30.105336 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:30.105487 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:30.144201 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:41997
I20250905 02:20:30.144299 486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:41997 every 8 connection(s)
I20250905 02:20:30.157624 487 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39045
I20250905 02:20:30.157931 487 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:30.158535 487 heartbeater.cc:507] Master 127.31.48.190:39045 requested a full tablet report, sending...
I20250905 02:20:30.160020 32501 ts_manager.cc:194] Registered new tserver with Master: 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132:41997)
I20250905 02:20:30.161309 32501 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41638
I20250905 02:20:31.163483 487 heartbeater.cc:499] Master 127.31.48.190:39045 was elected leader, sending a full tablet report...
I20250905 02:20:31.912815 32556 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:31.918514 32556 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:31.918927 32556 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:33.919757 32556 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:33.924912 32556 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:33.925244 32556 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:35.926120 32556 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:35.931430 32556 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:35.931808 32556 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:35.973524 32754 consensus_queue.cc:237] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } attrs { replace: true } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } }
I20250905 02:20:35.981313 32605 raft_consensus.cc:1273] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Refusing update from remote peer 4f5319a94bc54262a8ce9b7e5fea47fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:35.983325 32678 raft_consensus.cc:1273] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Refusing update from remote peer 4f5319a94bc54262a8ce9b7e5fea47fa: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:35.983247 502 consensus_queue.cc:1035] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 02:20:35.984941 501 consensus_queue.cc:1035] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:20:35.993594 502 raft_consensus.cc:2953] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } attrs { replace: true } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:35.994992 32605 raft_consensus.cc:2953] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } attrs { replace: true } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
W20250905 02:20:35.996773 32719 consensus_peers.cc:489] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa -> Peer 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132:41997): Couldn't send request to peer 6e05e6597ea1466e8d73ba147828c300. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 81675d685bdc4b149056a127137aa2d1. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:35.996099 32678 raft_consensus.cc:2953] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } attrs { replace: true } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.014000 32501 catalog_manager.cc:5582] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 reported cstate change: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } attrs { replace: true } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.024802 32605 consensus_queue.cc:237] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } }
I20250905 02:20:36.037367 32678 raft_consensus.cc:1273] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:36.039430 32754 raft_consensus.cc:1273] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:36.040580 497 consensus_queue.cc:1035] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 02:20:36.042429 497 consensus_queue.cc:1035] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:20:36.048224 32570 consensus_peers.cc:489] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 -> Peer 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132:41997): Couldn't send request to peer 6e05e6597ea1466e8d73ba147828c300. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cf808053b644930abd0e5fc8344cd2f. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.049893 515 raft_consensus.cc:2953] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.051306 32678 raft_consensus.cc:2953] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.053344 32754 raft_consensus.cc:2953] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.061496 32502 catalog_manager.cc:5582] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 reported cstate change: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.068830 32605 consensus_queue.cc:237] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } attrs { replace: true } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } }
W20250905 02:20:36.077232 32570 consensus_peers.cc:489] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 -> Peer 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132:41997): Couldn't send request to peer 6e05e6597ea1466e8d73ba147828c300. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f3dbbe24597498ab2a205e6cdf8d806. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.079814 32754 raft_consensus.cc:1273] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:36.080091 32678 raft_consensus.cc:1273] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:36.081346 515 consensus_queue.cc:1035] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:20:36.082165 518 consensus_queue.cc:1035] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 02:20:36.089478 515 raft_consensus.cc:2953] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } attrs { replace: true } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.091396 32754 raft_consensus.cc:2953] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } attrs { replace: true } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.091348 32678 raft_consensus.cc:2953] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } attrs { replace: true } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.103008 32502 catalog_manager.cc:5582] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 reported cstate change: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:36.105058 32605 consensus_queue.cc:237] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } }
I20250905 02:20:36.112315 32678 raft_consensus.cc:1273] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 2 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:20:36.112995 32754 raft_consensus.cc:1273] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 2 FOLLOWER]: Refusing update from remote peer 4ba0c6edbed6407bbb45011d4adc4d61: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:20:36.113513 518 consensus_queue.cc:1035] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:20:36.116061 518 consensus_queue.cc:1035] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:20:36.121642 32570 consensus_peers.cc:489] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 -> Peer 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132:41997): Couldn't send request to peer 6e05e6597ea1466e8d73ba147828c300. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61191ceca6324c52a4c2e7669736a8c1. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.125106 518 raft_consensus.cc:2953] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.126740 32678 raft_consensus.cc:2953] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.128273 32754 raft_consensus.cc:2953] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } } }
I20250905 02:20:36.136490 32501 catalog_manager.cc:5582] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 reported cstate change: config changed from index -1 to 2, NON_VOTER 6e05e6597ea1466e8d73ba147828c300 (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4ba0c6edbed6407bbb45011d4adc4d61" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 39891 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3156c20d9704874a9a53667636ce326" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 38253 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f5319a94bc54262a8ce9b7e5fea47fa" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e05e6597ea1466e8d73ba147828c300" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 41997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:36.179746 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
W20250905 02:20:36.189644 32490 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
W20250905 02:20:36.193169 32551 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:20:36.193650 32551 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:20:36.202790 32551 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:20:36.203034 32551 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:20:36.211019 32551 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:20:36.211321 32551 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250905 02:20:36.217255 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:36.218043 31938 tablet_replica.cc:331] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.219048 31938 raft_consensus.cc:2241] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.219509 31938 raft_consensus.cc:2270] T e5aba5dd131a405885e8666e93aed51d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.222018 31938 tablet_replica.cc:331] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.222693 31938 raft_consensus.cc:2241] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.223055 31938 raft_consensus.cc:2270] T 8958e6c9050d48a6af6fb9b333f7282d P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.224758 31938 tablet_replica.cc:331] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.225230 31938 raft_consensus.cc:2241] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.225610 31938 raft_consensus.cc:2270] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.227178 31938 tablet_replica.cc:331] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.227823 31938 raft_consensus.cc:2241] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.229207 31938 raft_consensus.cc:2270] T 61191ceca6324c52a4c2e7669736a8c1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 02:20:36.230253 32646 consensus_peers.cc:489] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 -> Peer 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891): Couldn't send request to peer 4ba0c6edbed6407bbb45011d4adc4d61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.231922 31938 tablet_replica.cc:331] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.232559 31938 raft_consensus.cc:2241] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.233039 31938 raft_consensus.cc:2270] T 7155cc6ee3994fc88cf33b4715eb443a P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.235332 31938 tablet_replica.cc:331] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.235867 31938 raft_consensus.cc:2241] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.236304 31938 raft_consensus.cc:2270] T 81675d685bdc4b149056a127137aa2d1 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.238968 31938 tablet_replica.cc:331] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.239571 31938 raft_consensus.cc:2241] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.240595 31938 raft_consensus.cc:2270] T 2cf808053b644930abd0e5fc8344cd2f P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.243173 31938 tablet_replica.cc:331] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61: stopping tablet replica
I20250905 02:20:36.243767 31938 raft_consensus.cc:2241] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.244877 31938 raft_consensus.cc:2270] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4ba0c6edbed6407bbb45011d4adc4d61 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 02:20:36.245987 32720 consensus_peers.cc:489] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa -> Peer 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891): Couldn't send request to peer 4ba0c6edbed6407bbb45011d4adc4d61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.277462 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:20:36.297451 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
W20250905 02:20:36.320542 32646 consensus_peers.cc:489] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 -> Peer 4ba0c6edbed6407bbb45011d4adc4d61 (127.31.48.129:39891): Couldn't send request to peer 4ba0c6edbed6407bbb45011d4adc4d61. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:39891: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.336778 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:36.337560 31938 tablet_replica.cc:331] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.338495 31938 raft_consensus.cc:2241] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.339591 31938 raft_consensus.cc:2270] T e5aba5dd131a405885e8666e93aed51d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.341634 31938 tablet_replica.cc:331] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.342202 31938 raft_consensus.cc:2241] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.342643 31938 raft_consensus.cc:2270] T 61191ceca6324c52a4c2e7669736a8c1 P d3156c20d9704874a9a53667636ce326 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.345170 31938 tablet_replica.cc:331] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.345800 31938 raft_consensus.cc:2241] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.346505 31938 raft_consensus.cc:2270] T 7155cc6ee3994fc88cf33b4715eb443a P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.348553 31938 tablet_replica.cc:331] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.349076 31938 raft_consensus.cc:2241] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.349459 31938 raft_consensus.cc:2270] T 8958e6c9050d48a6af6fb9b333f7282d P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.351900 31938 tablet_replica.cc:331] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.352418 31938 raft_consensus.cc:2241] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.352975 31938 raft_consensus.cc:2270] T 81675d685bdc4b149056a127137aa2d1 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.355456 31938 tablet_replica.cc:331] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.356079 31938 raft_consensus.cc:2241] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.356535 31938 raft_consensus.cc:2270] T 4f3dbbe24597498ab2a205e6cdf8d806 P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.359822 31938 tablet_replica.cc:331] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.360306 31938 raft_consensus.cc:2241] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.360823 31938 raft_consensus.cc:2270] T 2cf808053b644930abd0e5fc8344cd2f P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.363133 31938 tablet_replica.cc:331] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326: stopping tablet replica
I20250905 02:20:36.363826 31938 raft_consensus.cc:2241] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.364451 31938 raft_consensus.cc:2270] T d5cfb0a9668b47a7ab7437e0dabae0ee P d3156c20d9704874a9a53667636ce326 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.369558 530 ts_tablet_manager.cc:927] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: Initiating tablet copy from peer 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609)
I20250905 02:20:36.371749 530 tablet_copy_client.cc:323] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.131:41609
I20250905 02:20:36.383253 32764 tablet_copy_service.cc:140] P 4f5319a94bc54262a8ce9b7e5fea47fa: Received BeginTabletCopySession request for tablet 81675d685bdc4b149056a127137aa2d1 from peer 6e05e6597ea1466e8d73ba147828c300 ({username='slave'} at 127.0.0.1:41318)
I20250905 02:20:36.383808 32764 tablet_copy_service.cc:161] P 4f5319a94bc54262a8ce9b7e5fea47fa: Beginning new tablet copy session on tablet 81675d685bdc4b149056a127137aa2d1 from peer 6e05e6597ea1466e8d73ba147828c300 at {username='slave'} at 127.0.0.1:41318: session id = 6e05e6597ea1466e8d73ba147828c300-81675d685bdc4b149056a127137aa2d1
I20250905 02:20:36.389508 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
W20250905 02:20:36.390591 32719 consensus_peers.cc:489] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa -> Peer d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253): Couldn't send request to peer d3156c20d9704874a9a53667636ce326. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.130:38253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.391444 32764 tablet_copy_source_session.cc:215] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:20:36.395411 530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81675d685bdc4b149056a127137aa2d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:36.414333 530 tablet_copy_client.cc:806] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: tablet copy: Starting download of 0 data blocks...
I20250905 02:20:36.415110 530 tablet_copy_client.cc:670] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: tablet copy: Starting download of 1 WAL segments...
I20250905 02:20:36.416453 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
W20250905 02:20:36.422899 32551 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:20:36.423153 32551 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20250905 02:20:36.429013 31938 tablet_copy_service.cc:362] P 4f5319a94bc54262a8ce9b7e5fea47fa: Destroying tablet copy session 6e05e6597ea1466e8d73ba147828c300-81675d685bdc4b149056a127137aa2d1 due to service shutdown
I20250905 02:20:36.429399 31938 tablet_copy_service.cc:434] P 4f5319a94bc54262a8ce9b7e5fea47fa: ending tablet copy session 6e05e6597ea1466e8d73ba147828c300-81675d685bdc4b149056a127137aa2d1 on tablet 81675d685bdc4b149056a127137aa2d1 with peer 6e05e6597ea1466e8d73ba147828c300
I20250905 02:20:36.442335 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:36.442915 31938 tablet_replica.cc:331] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.443406 31938 raft_consensus.cc:2241] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.443774 31938 raft_consensus.cc:2270] T e5aba5dd131a405885e8666e93aed51d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.445389 31938 tablet_replica.cc:331] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.445817 31938 raft_consensus.cc:2241] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.446445 31938 raft_consensus.cc:2270] T d5cfb0a9668b47a7ab7437e0dabae0ee P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.448025 31938 tablet_replica.cc:331] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.448531 31938 raft_consensus.cc:2241] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.448927 31938 raft_consensus.cc:2270] T 7155cc6ee3994fc88cf33b4715eb443a P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.450834 31938 tablet_replica.cc:331] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.451313 31938 raft_consensus.cc:2241] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.451717 31938 raft_consensus.cc:2270] T 61191ceca6324c52a4c2e7669736a8c1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 02:20:36.453753 32719 consensus_peers.cc:489] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa -> Peer d3156c20d9704874a9a53667636ce326 (127.31.48.130:38253): Couldn't send request to peer d3156c20d9704874a9a53667636ce326. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.130:38253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:36.454247 31938 tablet_replica.cc:331] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.454768 31938 raft_consensus.cc:2241] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.455581 31938 raft_consensus.cc:2270] T 81675d685bdc4b149056a127137aa2d1 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.457561 31938 tablet_replica.cc:331] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.458048 31938 raft_consensus.cc:2241] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.458448 31938 raft_consensus.cc:2270] T 4f3dbbe24597498ab2a205e6cdf8d806 P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.460690 31938 tablet_replica.cc:331] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.461297 31938 raft_consensus.cc:2241] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:36.461709 31938 raft_consensus.cc:2270] T 2cf808053b644930abd0e5fc8344cd2f P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.463603 31938 tablet_replica.cc:331] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa: stopping tablet replica
I20250905 02:20:36.464047 31938 raft_consensus.cc:2241] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.464802 31938 raft_consensus.cc:2270] T 8958e6c9050d48a6af6fb9b333f7282d P 4f5319a94bc54262a8ce9b7e5fea47fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.490353 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:20:36.505452 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:20:36.524518 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20250905 02:20:36.549714 530 ts_tablet_manager.cc:1005] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: Tablet Copy: Unable to fetch data from remote peer 4f5319a94bc54262a8ce9b7e5fea47fa (127.31.48.131:41609): Network error: Unable to download WAL segment with seq. number 1: unable to fetch data from remote: Client connection negotiation failed: client connection to 127.31.48.131:41609: connect: Connection refused (error 111)
I20250905 02:20:36.550076 530 tablet_replica.cc:331] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: stopping tablet replica
I20250905 02:20:36.550345 530 raft_consensus.cc:2241] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300 [term 1 LEARNER]: Raft consensus shutting down.
I20250905 02:20:36.550783 530 raft_consensus.cc:2270] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300 [term 1 LEARNER]: Raft consensus is shut down!
W20250905 02:20:36.552835 530 tablet_copy_client.cc:1131] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: Client connection negotiation failed: client connection to 127.31.48.131:41609: connect: Connection refused (error 111)
W20250905 02:20:36.553911 530 tablet_metadata.cc:476] failed to load DataDirGroup from superblock: Already present: tried to load directory group for tablet 81675d685bdc4b149056a127137aa2d1 but one is already registered
I20250905 02:20:36.554261 530 ts_tablet_manager.cc:1905] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:20:36.563222 530 ts_tablet_manager.cc:1918] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.0
I20250905 02:20:36.563506 530 log.cc:1199] T 81675d685bdc4b149056a127137aa2d1 P 6e05e6597ea1466e8d73ba147828c300: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingTurnOffAndOn.1757038820745333-31938-0/minicluster-data/ts-3-root/wals/81675d685bdc4b149056a127137aa2d1
I20250905 02:20:36.580282 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:20:36.587779 31938 master.cc:561] Master@127.31.48.190:39045 shutting down...
I20250905 02:20:36.603704 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:36.604178 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:36.604511 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P d4d77197926e4fde9ce8c7cf1b51a02d: stopping tablet replica
I20250905 02:20:36.623317 31938 master.cc:583] Master@127.31.48.190:39045 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingTurnOffAndOn (10870 ms)
[ RUN ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing
I20250905 02:20:36.656993 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:36367,127.31.48.189:38657,127.31.48.188:42595
I20250905 02:20:36.658252 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:36.663713 534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.664090 533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.664158 536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:36.665845 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:36.666538 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:36.666716 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:36.666831 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038836666820 us; error 0 us; skew 500 ppm
I20250905 02:20:36.667261 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:36.669445 31938 webserver.cc:480] Webserver started at http://127.31.48.190:38423/ using document root <none> and password file <none>
I20250905 02:20:36.669852 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:36.670028 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:36.670229 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:36.671243 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "2a9e035803f04da19fae285aa889e23d"
format_stamp: "Formatted at 2025-09-05 02:20:36 on dist-test-slave-q32q"
I20250905 02:20:36.675470 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 02:20:36.678607 541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:36.679337 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:20:36.679569 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "2a9e035803f04da19fae285aa889e23d"
format_stamp: "Formatted at 2025-09-05 02:20:36 on dist-test-slave-q32q"
I20250905 02:20:36.679791 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:36.699231 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:36.700314 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:36.735581 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:36367
I20250905 02:20:36.735671 592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:36367 every 8 connection(s)
I20250905 02:20:36.739039 593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:36.739181 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:36.743970 595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.744726 596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.746563 598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:36.746491 593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:36.747231 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:36.748059 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:36.748332 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:36.748564 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038836748546 us; error 0 us; skew 500 ppm
I20250905 02:20:36.749363 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:36.751935 31938 webserver.cc:480] Webserver started at http://127.31.48.189:38883/ using document root <none> and password file <none>
I20250905 02:20:36.752588 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:36.752825 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:36.753068 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:36.754010 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root/instance:
uuid: "c39fe298fa9d4bbdb438c0799cce20a9"
format_stamp: "Formatted at 2025-09-05 02:20:36 on dist-test-slave-q32q"
I20250905 02:20:36.759327 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250905 02:20:36.759632 593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:36.763388 606 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:36.764173 31938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250905 02:20:36.764453 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root
uuid: "c39fe298fa9d4bbdb438c0799cce20a9"
format_stamp: "Formatted at 2025-09-05 02:20:36 on dist-test-slave-q32q"
W20250905 02:20:36.764829 593 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.189:38657: Network error: Client connection negotiation failed: client connection to 127.31.48.189:38657: connect: Connection refused (error 111)
I20250905 02:20:36.764732 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:36.778801 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:36.779989 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:36.798345 593 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } attempt: 1
W20250905 02:20:36.803701 593 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.189:38657: Network error: Client connection negotiation failed: client connection to 127.31.48.189:38657: connect: Connection refused (error 111)
I20250905 02:20:36.817608 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.189:38657
I20250905 02:20:36.817705 657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.189:38657 every 8 connection(s)
I20250905 02:20:36.821201 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:36.821331 658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 02:20:36.826134 660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:36.827267 661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:36.829885 658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:36.830482 663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:36.830910 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:36.832082 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:36.832342 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:36.832566 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038836832544 us; error 0 us; skew 500 ppm
I20250905 02:20:36.833047 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:36.840113 31938 webserver.cc:480] Webserver started at http://127.31.48.188:46627/ using document root <none> and password file <none>
I20250905 02:20:36.840637 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:36.840848 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:36.841104 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:36.842120 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root/instance:
uuid: "353be9287593454abec1e3bd6c5dd266"
format_stamp: "Formatted at 2025-09-05 02:20:36 on dist-test-slave-q32q"
I20250905 02:20:36.844197 593 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } attempt: 2
I20250905 02:20:36.847693 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250905 02:20:36.848577 658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:36.855242 671 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:36.856408 31938 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.004s
I20250905 02:20:36.856724 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root
uuid: "353be9287593454abec1e3bd6c5dd266"
format_stamp: "Formatted at 2025-09-05 02:20:36 on dist-test-slave-q32q"
I20250905 02:20:36.857100 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/master-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:36.858162 593 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:36.865933 593 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:42595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:42595: connect: Connection refused (error 111)
I20250905 02:20:36.866463 658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
W20250905 02:20:36.871429 658 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:42595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:42595: connect: Connection refused (error 111)
I20250905 02:20:36.883281 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:36.884487 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:36.902465 593 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } attempt: 1
W20250905 02:20:36.907644 593 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:42595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:42595: connect: Connection refused (error 111)
I20250905 02:20:36.910961 658 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } attempt: 1
W20250905 02:20:36.916054 658 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.31.48.188:42595: Network error: Client connection negotiation failed: client connection to 127.31.48.188:42595: connect: Connection refused (error 111)
I20250905 02:20:36.923141 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.188:42595
I20250905 02:20:36.923233 724 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.188:42595 every 8 connection(s)
I20250905 02:20:36.925989 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:20:36.926944 725 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:36.932560 725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:36.942538 725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:36.951716 725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } has no permanent_uuid. Determining permanent_uuid...
I20250905 02:20:36.955266 593 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } attempt: 2
I20250905 02:20:36.968029 725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: Bootstrap starting.
I20250905 02:20:36.970093 593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d: Bootstrap starting.
I20250905 02:20:36.972611 725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:36.974422 593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:36.976814 725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: No bootstrap required, opened a new log
I20250905 02:20:36.978727 593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d: No bootstrap required, opened a new log
I20250905 02:20:36.979084 725 raft_consensus.cc:357] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:36.979568 725 raft_consensus.cc:383] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:36.979808 725 raft_consensus.cc:738] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353be9287593454abec1e3bd6c5dd266, State: Initialized, Role: FOLLOWER
I20250905 02:20:36.980619 658 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } attempt: 2
I20250905 02:20:36.980728 593 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:36.980334 725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:36.981349 593 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:36.981755 593 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9e035803f04da19fae285aa889e23d, State: Initialized, Role: FOLLOWER
I20250905 02:20:36.982522 593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:36.984385 732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:36.985483 732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:36.985925 725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:36.993136 593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:36.991077 733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:36.994405 733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:37.003276 658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: Bootstrap starting.
W20250905 02:20:37.005281 745 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 02:20:37.005618 745 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 02:20:37.008111 658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:37.011022 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 1
I20250905 02:20:37.012598 658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: No bootstrap required, opened a new log
W20250905 02:20:37.013834 756 catalog_manager.cc:1560] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 02:20:37.014092 756 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 02:20:37.014675 658 raft_consensus.cc:357] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:37.015197 658 raft_consensus.cc:383] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:37.015389 658 raft_consensus.cc:738] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39fe298fa9d4bbdb438c0799cce20a9, State: Initialized, Role: FOLLOWER
I20250905 02:20:37.015971 658 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:37.017493 757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.018002 757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:37.018627 658 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:37.028326 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 2
I20250905 02:20:37.029726 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:37.031378 768 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 02:20:37.031673 768 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250905 02:20:37.036145 769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:37.036321 770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:37.039346 772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:37.041647 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:37.042405 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:37.042598 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:37.042748 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038837042736 us; error 0 us; skew 500 ppm
I20250905 02:20:37.043164 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:37.045498 31938 webserver.cc:480] Webserver started at http://127.31.48.129:39907/ using document root <none> and password file <none>
I20250905 02:20:37.045934 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:37.046108 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:37.046315 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:37.047389 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "ca8c10e0f7b34e4d9c6288037010fa0d"
format_stamp: "Formatted at 2025-09-05 02:20:37 on dist-test-slave-q32q"
I20250905 02:20:37.051684 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:20:37.055115 777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.055848 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:20:37.056068 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "ca8c10e0f7b34e4d9c6288037010fa0d"
format_stamp: "Formatted at 2025-09-05 02:20:37 on dist-test-slave-q32q"
I20250905 02:20:37.056277 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:37.078847 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:37.079942 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:37.081353 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:37.083580 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:37.083781 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.084028 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:37.084220 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.121968 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:40155
I20250905 02:20:37.122057 839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:40155 every 8 connection(s)
I20250905 02:20:37.124891 733 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:37.125545 733 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:37.128068 733 leader_election.cc:290] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39fe298fa9d4bbdb438c0799cce20a9 (127.31.48.189:38657), 353be9287593454abec1e3bd6c5dd266 (127.31.48.188:42595)
I20250905 02:20:37.129317 633 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e035803f04da19fae285aa889e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" is_pre_election: true
I20250905 02:20:37.130245 633 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9e035803f04da19fae285aa889e23d in term 0.
I20250905 02:20:37.131911 543 leader_election.cc:304] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9e035803f04da19fae285aa889e23d, c39fe298fa9d4bbdb438c0799cce20a9; no voters:
I20250905 02:20:37.132190 700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e035803f04da19fae285aa889e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353be9287593454abec1e3bd6c5dd266" is_pre_election: true
I20250905 02:20:37.133016 700 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9e035803f04da19fae285aa889e23d in term 0.
I20250905 02:20:37.136816 733 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:37.137429 733 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:37.137899 733 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:37.147411 733 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:37.151594 633 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e035803f04da19fae285aa889e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39fe298fa9d4bbdb438c0799cce20a9"
I20250905 02:20:37.152223 633 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:37.152101 700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9e035803f04da19fae285aa889e23d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "353be9287593454abec1e3bd6c5dd266"
I20250905 02:20:37.153237 700 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:37.161213 633 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9e035803f04da19fae285aa889e23d in term 1.
I20250905 02:20:37.162499 543 leader_election.cc:304] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9e035803f04da19fae285aa889e23d, c39fe298fa9d4bbdb438c0799cce20a9; no voters:
I20250905 02:20:37.165367 841 heartbeater.cc:344] Connected to a master server at 127.31.48.189:38657
I20250905 02:20:37.165753 841 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.166651 841 heartbeater.cc:507] Master 127.31.48.189:38657 requested a full tablet report, sending...
I20250905 02:20:37.161810 733 leader_election.cc:290] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [CANDIDATE]: Term 1 election: Requested vote from peers c39fe298fa9d4bbdb438c0799cce20a9 (127.31.48.189:38657), 353be9287593454abec1e3bd6c5dd266 (127.31.48.188:42595)
I20250905 02:20:37.170012 623 ts_manager.cc:194] Registered new tserver with Master: ca8c10e0f7b34e4d9c6288037010fa0d (127.31.48.129:40155)
I20250905 02:20:37.172173 733 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:37.172677 733 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 1 LEADER]: Becoming Leader. State: Replica: 2a9e035803f04da19fae285aa889e23d, State: Running, Role: LEADER
I20250905 02:20:37.172820 840 heartbeater.cc:344] Connected to a master server at 127.31.48.188:42595
I20250905 02:20:37.173379 840 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.173556 733 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:37.174439 840 heartbeater.cc:507] Master 127.31.48.188:42595 requested a full tablet report, sending...
I20250905 02:20:37.175734 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:37.178272 700 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9e035803f04da19fae285aa889e23d in term 1.
I20250905 02:20:37.179682 842 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36367
I20250905 02:20:37.180104 842 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.180388 690 ts_manager.cc:194] Registered new tserver with Master: ca8c10e0f7b34e4d9c6288037010fa0d (127.31.48.129:40155)
I20250905 02:20:37.181087 842 heartbeater.cc:507] Master 127.31.48.190:36367 requested a full tablet report, sending...
I20250905 02:20:37.182821 847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9e035803f04da19fae285aa889e23d. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.183547 847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:37.183662 558 ts_manager.cc:194] Registered new tserver with Master: ca8c10e0f7b34e4d9c6288037010fa0d (127.31.48.129:40155)
I20250905 02:20:37.185281 849 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
W20250905 02:20:37.188449 850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:37.189294 851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:37.192173 853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:37.192425 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:37.192600 849 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:37.193387 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:37.193588 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:37.193766 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038837193748 us; error 0 us; skew 500 ppm
I20250905 02:20:37.194306 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:37.197075 31938 webserver.cc:480] Webserver started at http://127.31.48.130:36253/ using document root <none> and password file <none>
I20250905 02:20:37.197503 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:37.197666 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:37.197921 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:37.198940 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "b4ab8938afac48798dfcc396fa86bd7a"
format_stamp: "Formatted at 2025-09-05 02:20:37 on dist-test-slave-q32q"
I20250905 02:20:37.200649 633 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Refusing update from remote peer 2a9e035803f04da19fae285aa889e23d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:37.201198 700 raft_consensus.cc:1273] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 1 FOLLOWER]: Refusing update from remote peer 2a9e035803f04da19fae285aa889e23d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:37.201812 847 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:37.202531 733 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [LEADER]: Connected to new peer: Peer: permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:37.205114 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.007s
I20250905 02:20:37.212081 732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9e035803f04da19fae285aa889e23d. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.212862 732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:37.213764 862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.214581 31938 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.004s
I20250905 02:20:37.214879 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "b4ab8938afac48798dfcc396fa86bd7a"
format_stamp: "Formatted at 2025-09-05 02:20:37 on dist-test-slave-q32q"
I20250905 02:20:37.215137 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:37.214947 757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9e035803f04da19fae285aa889e23d. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.215714 757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:37.223981 732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.224949 732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:37.227312 733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.228042 733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:37.231467 757 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.232185 757 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 02:20:37.231945 849 catalog_manager.cc:1349] Generated new cluster ID: a2a053e889d248cb92a9a5a9af8cf3c6
I20250905 02:20:37.232824 849 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:37.234239 869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9e035803f04da19fae285aa889e23d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:37.235100 869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:37.241252 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:37.242381 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:37.243846 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:37.246261 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:37.246441 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.246649 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:37.246820 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.256275 849 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:37.258433 849 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:37.285585 849 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d: Generated new TSK 0
I20250905 02:20:37.286509 849 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:37.310259 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:44453
I20250905 02:20:37.310351 929 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:44453 every 8 connection(s)
I20250905 02:20:37.321627 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:20:37.332763 930 heartbeater.cc:344] Connected to a master server at 127.31.48.188:42595
I20250905 02:20:37.333403 930 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.334600 930 heartbeater.cc:507] Master 127.31.48.188:42595 requested a full tablet report, sending...
I20250905 02:20:37.338547 690 ts_manager.cc:194] Registered new tserver with Master: b4ab8938afac48798dfcc396fa86bd7a (127.31.48.130:44453)
I20250905 02:20:37.338562 931 heartbeater.cc:344] Connected to a master server at 127.31.48.189:38657
I20250905 02:20:37.341287 931 heartbeater.cc:461] Registering TS with master...
W20250905 02:20:37.341663 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:37.342793 931 heartbeater.cc:507] Master 127.31.48.189:38657 requested a full tablet report, sending...
W20250905 02:20:37.343892 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:37.345829 932 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36367
I20250905 02:20:37.346184 932 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.346983 932 heartbeater.cc:507] Master 127.31.48.190:36367 requested a full tablet report, sending...
I20250905 02:20:37.347309 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:37.347352 623 ts_manager.cc:194] Registered new tserver with Master: b4ab8938afac48798dfcc396fa86bd7a (127.31.48.130:44453)
W20250905 02:20:37.348011 941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:37.349082 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 02:20:37.349090 558 ts_manager.cc:194] Registered new tserver with Master: b4ab8938afac48798dfcc396fa86bd7a (127.31.48.130:44453)
W20250905 02:20:37.349363 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:37.349572 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038837349560 us; error 0 us; skew 500 ppm
I20250905 02:20:37.350067 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:37.350970 558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57506
I20250905 02:20:37.352691 31938 webserver.cc:480] Webserver started at http://127.31.48.131:44121/ using document root <none> and password file <none>
I20250905 02:20:37.353296 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:37.353520 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:37.353725 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:37.354590 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "dab4364aa92f454194e281ca2a9bd753"
format_stamp: "Formatted at 2025-09-05 02:20:37 on dist-test-slave-q32q"
I20250905 02:20:37.358518 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 02:20:37.361582 945 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.362367 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:20:37.362641 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "dab4364aa92f454194e281ca2a9bd753"
format_stamp: "Formatted at 2025-09-05 02:20:37 on dist-test-slave-q32q"
I20250905 02:20:37.362910 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NextLeaderResumesAutoRebalancing.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:37.377780 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:37.378913 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:37.380252 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:37.382436 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:37.382640 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.382864 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:37.383026 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:37.421815 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:33333
I20250905 02:20:37.421919 1007 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:33333 every 8 connection(s)
I20250905 02:20:37.452291 1008 heartbeater.cc:344] Connected to a master server at 127.31.48.188:42595
I20250905 02:20:37.452780 1008 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.453718 1008 heartbeater.cc:507] Master 127.31.48.188:42595 requested a full tablet report, sending...
I20250905 02:20:37.455518 1009 heartbeater.cc:344] Connected to a master server at 127.31.48.189:38657
I20250905 02:20:37.455853 1009 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.456756 1009 heartbeater.cc:507] Master 127.31.48.189:38657 requested a full tablet report, sending...
I20250905 02:20:37.456907 690 ts_manager.cc:194] Registered new tserver with Master: dab4364aa92f454194e281ca2a9bd753 (127.31.48.131:33333)
I20250905 02:20:37.458730 1010 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36367
I20250905 02:20:37.459105 1010 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:37.459642 623 ts_manager.cc:194] Registered new tserver with Master: dab4364aa92f454194e281ca2a9bd753 (127.31.48.131:33333)
I20250905 02:20:37.459880 1010 heartbeater.cc:507] Master 127.31.48.190:36367 requested a full tablet report, sending...
I20250905 02:20:37.461716 558 ts_manager.cc:194] Registered new tserver with Master: dab4364aa92f454194e281ca2a9bd753 (127.31.48.131:33333)
I20250905 02:20:37.461871 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015890748s
I20250905 02:20:37.463400 558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57520
I20250905 02:20:38.009306 745 catalog_manager.cc:1261] Loaded cluster ID: a2a053e889d248cb92a9a5a9af8cf3c6
I20250905 02:20:38.009588 745 catalog_manager.cc:1554] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: loading cluster ID for follower catalog manager: success
I20250905 02:20:38.013787 745 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: acquiring CA information for follower catalog manager: success
I20250905 02:20:38.017095 745 catalog_manager.cc:1604] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 02:20:38.035298 768 catalog_manager.cc:1261] Loaded cluster ID: a2a053e889d248cb92a9a5a9af8cf3c6
I20250905 02:20:38.035573 768 catalog_manager.cc:1554] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: loading cluster ID for follower catalog manager: success
I20250905 02:20:38.039816 768 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: acquiring CA information for follower catalog manager: success
I20250905 02:20:38.043002 768 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 02:20:38.188048 558 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57498
I20250905 02:20:38.353686 932 heartbeater.cc:499] Master 127.31.48.190:36367 was elected leader, sending a full tablet report...
I20250905 02:20:38.465927 1010 heartbeater.cc:499] Master 127.31.48.190:36367 was elected leader, sending a full tablet report...
I20250905 02:20:39.010710 754 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:39.011946 31938 test_util.cc:276] Using random seed: 2077956692
I20250905 02:20:39.055374 558 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57530:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 02:20:39.122177 895 tablet_service.cc:1468] Processing CreateTablet for tablet d7c58f1a395b4581a7196f5a9d83ba15 (DEFAULT_TABLE table=test-workload [id=149190314275419aaf8b48b28614d764]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250905 02:20:39.123579 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7c58f1a395b4581a7196f5a9d83ba15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:39.127862 805 tablet_service.cc:1468] Processing CreateTablet for tablet a7fd3bf8d7764d20ba07a1cbb94c394b (DEFAULT_TABLE table=test-workload [id=149190314275419aaf8b48b28614d764]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250905 02:20:39.128409 973 tablet_service.cc:1468] Processing CreateTablet for tablet 77703afd391b469f89cb646f62c0e569 (DEFAULT_TABLE table=test-workload [id=149190314275419aaf8b48b28614d764]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250905 02:20:39.129166 805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7fd3bf8d7764d20ba07a1cbb94c394b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:39.129444 973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77703afd391b469f89cb646f62c0e569. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:39.139331 1044 tablet_bootstrap.cc:492] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a: Bootstrap starting.
I20250905 02:20:39.141086 1045 tablet_bootstrap.cc:492] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753: Bootstrap starting.
I20250905 02:20:39.145458 1046 tablet_bootstrap.cc:492] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d: Bootstrap starting.
I20250905 02:20:39.147224 1045 tablet_bootstrap.cc:654] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:39.147257 1044 tablet_bootstrap.cc:654] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:39.151782 1045 tablet_bootstrap.cc:492] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753: No bootstrap required, opened a new log
I20250905 02:20:39.152232 1045 ts_tablet_manager.cc:1397] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20250905 02:20:39.152840 1046 tablet_bootstrap.cc:654] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:39.154552 1044 tablet_bootstrap.cc:492] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a: No bootstrap required, opened a new log
I20250905 02:20:39.155068 1044 ts_tablet_manager.cc:1397] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.002s
I20250905 02:20:39.155006 1045 raft_consensus.cc:357] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4364aa92f454194e281ca2a9bd753" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33333 } }
I20250905 02:20:39.155743 1045 raft_consensus.cc:383] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:39.156105 1045 raft_consensus.cc:738] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab4364aa92f454194e281ca2a9bd753, State: Initialized, Role: FOLLOWER
I20250905 02:20:39.156818 1045 consensus_queue.cc:260] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4364aa92f454194e281ca2a9bd753" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33333 } }
I20250905 02:20:39.157554 1045 raft_consensus.cc:397] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:39.157866 1045 raft_consensus.cc:491] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:39.158238 1045 raft_consensus.cc:3058] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:39.158166 1044 raft_consensus.cc:357] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8938afac48798dfcc396fa86bd7a" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44453 } }
I20250905 02:20:39.158874 1044 raft_consensus.cc:383] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:39.159128 1046 tablet_bootstrap.cc:492] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d: No bootstrap required, opened a new log
I20250905 02:20:39.159188 1044 raft_consensus.cc:738] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ab8938afac48798dfcc396fa86bd7a, State: Initialized, Role: FOLLOWER
I20250905 02:20:39.159605 1046 ts_tablet_manager.cc:1397] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20250905 02:20:39.159967 1044 consensus_queue.cc:260] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8938afac48798dfcc396fa86bd7a" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44453 } }
I20250905 02:20:39.160662 1044 raft_consensus.cc:397] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:39.160952 1044 raft_consensus.cc:491] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:39.161284 1044 raft_consensus.cc:3058] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:39.161903 1046 raft_consensus.cc:357] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8c10e0f7b34e4d9c6288037010fa0d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 40155 } }
I20250905 02:20:39.162330 1046 raft_consensus.cc:383] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:39.162616 1046 raft_consensus.cc:738] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca8c10e0f7b34e4d9c6288037010fa0d, State: Initialized, Role: FOLLOWER
I20250905 02:20:39.163095 1046 consensus_queue.cc:260] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8c10e0f7b34e4d9c6288037010fa0d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 40155 } }
I20250905 02:20:39.163518 1046 raft_consensus.cc:397] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:39.163724 1046 raft_consensus.cc:491] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:39.163967 1046 raft_consensus.cc:3058] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:39.165025 1045 raft_consensus.cc:513] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4364aa92f454194e281ca2a9bd753" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33333 } }
I20250905 02:20:39.165759 1045 leader_election.cc:304] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dab4364aa92f454194e281ca2a9bd753; no voters:
I20250905 02:20:39.167245 1045 leader_election.cc:290] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:39.167631 1050 raft_consensus.cc:2802] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:39.168107 1044 raft_consensus.cc:513] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8938afac48798dfcc396fa86bd7a" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44453 } }
I20250905 02:20:39.169212 1044 leader_election.cc:304] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ab8938afac48798dfcc396fa86bd7a; no voters:
I20250905 02:20:39.169402 1046 raft_consensus.cc:513] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8c10e0f7b34e4d9c6288037010fa0d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 40155 } }
I20250905 02:20:39.170281 1046 leader_election.cc:304] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca8c10e0f7b34e4d9c6288037010fa0d; no voters:
I20250905 02:20:39.173537 1050 raft_consensus.cc:695] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 1 LEADER]: Becoming Leader. State: Replica: dab4364aa92f454194e281ca2a9bd753, State: Running, Role: LEADER
I20250905 02:20:39.174806 1050 consensus_queue.cc:237] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4364aa92f454194e281ca2a9bd753" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33333 } }
I20250905 02:20:39.191150 842 heartbeater.cc:499] Master 127.31.48.190:36367 was elected leader, sending a full tablet report...
I20250905 02:20:39.197817 558 catalog_manager.cc:5582] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 reported cstate change: term changed from 0 to 1, leader changed from <none> to dab4364aa92f454194e281ca2a9bd753 (127.31.48.131). New cstate: current_term: 1 leader_uuid: "dab4364aa92f454194e281ca2a9bd753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dab4364aa92f454194e281ca2a9bd753" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33333 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:39.258580 1044 leader_election.cc:290] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:39.259860 1045 ts_tablet_manager.cc:1428] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753: Time spent starting tablet: real 0.107s user 0.030s sys 0.061s
I20250905 02:20:39.260488 1052 raft_consensus.cc:2802] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:39.261924 1046 leader_election.cc:290] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:39.266589 1044 ts_tablet_manager.cc:1428] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a: Time spent starting tablet: real 0.111s user 0.045s sys 0.053s
I20250905 02:20:39.267230 1052 raft_consensus.cc:695] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 1 LEADER]: Becoming Leader. State: Replica: b4ab8938afac48798dfcc396fa86bd7a, State: Running, Role: LEADER
I20250905 02:20:39.268082 1052 consensus_queue.cc:237] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8938afac48798dfcc396fa86bd7a" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44453 } }
I20250905 02:20:39.269098 1046 ts_tablet_manager.cc:1428] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d: Time spent starting tablet: real 0.109s user 0.038s sys 0.050s
I20250905 02:20:39.270674 1053 raft_consensus.cc:2802] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:39.283948 1053 raft_consensus.cc:695] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 1 LEADER]: Becoming Leader. State: Replica: ca8c10e0f7b34e4d9c6288037010fa0d, State: Running, Role: LEADER
I20250905 02:20:39.284754 1053 consensus_queue.cc:237] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8c10e0f7b34e4d9c6288037010fa0d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 40155 } }
I20250905 02:20:39.288324 556 catalog_manager.cc:5582] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a reported cstate change: term changed from 0 to 1, leader changed from <none> to b4ab8938afac48798dfcc396fa86bd7a (127.31.48.130). New cstate: current_term: 1 leader_uuid: "b4ab8938afac48798dfcc396fa86bd7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8938afac48798dfcc396fa86bd7a" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44453 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:39.303182 557 catalog_manager.cc:5582] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d reported cstate change: term changed from 0 to 1, leader changed from <none> to ca8c10e0f7b34e4d9c6288037010fa0d (127.31.48.129). New cstate: current_term: 1 leader_uuid: "ca8c10e0f7b34e4d9c6288037010fa0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca8c10e0f7b34e4d9c6288037010fa0d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 40155 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:39.354735 31938 master.cc:561] Master@127.31.48.190:36367 shutting down...
I20250905 02:20:39.369390 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:39.370205 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:39.370520 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2a9e035803f04da19fae285aa889e23d: stopping tablet replica
I20250905 02:20:39.390738 31938 master.cc:583] Master@127.31.48.190:36367 shutdown complete.
W20250905 02:20:40.306545 1010 heartbeater.cc:646] Failed to heartbeat to 127.31.48.190:36367 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.48.190:36367: connect: Connection refused (error 111)
I20250905 02:20:40.829594 1068 raft_consensus.cc:491] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a9e035803f04da19fae285aa889e23d)
I20250905 02:20:40.830269 1068 raft_consensus.cc:513] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:40.830690 1069 raft_consensus.cc:491] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a9e035803f04da19fae285aa889e23d)
I20250905 02:20:40.831117 1069 raft_consensus.cc:513] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:40.832038 1068 leader_election.cc:290] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a9e035803f04da19fae285aa889e23d (127.31.48.190:36367), 353be9287593454abec1e3bd6c5dd266 (127.31.48.188:42595)
I20250905 02:20:40.835359 700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "353be9287593454abec1e3bd6c5dd266" is_pre_election: true
I20250905 02:20:40.836053 700 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39fe298fa9d4bbdb438c0799cce20a9 in term 1.
I20250905 02:20:40.837473 609 leader_election.cc:304] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 353be9287593454abec1e3bd6c5dd266, c39fe298fa9d4bbdb438c0799cce20a9; no voters:
I20250905 02:20:40.838462 1068 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:20:40.838824 1068 raft_consensus.cc:491] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2a9e035803f04da19fae285aa889e23d)
I20250905 02:20:40.839175 1068 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 1 FOLLOWER]: Advancing to term 2
W20250905 02:20:40.840451 610 leader_election.cc:336] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a9e035803f04da19fae285aa889e23d (127.31.48.190:36367): Network error: Client connection negotiation failed: client connection to 127.31.48.190:36367: connect: Connection refused (error 111)
I20250905 02:20:40.841392 632 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "353be9287593454abec1e3bd6c5dd266" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" is_pre_election: true
I20250905 02:20:40.842543 1069 leader_election.cc:290] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a9e035803f04da19fae285aa889e23d (127.31.48.190:36367), c39fe298fa9d4bbdb438c0799cce20a9 (127.31.48.189:38657)
W20250905 02:20:40.844594 676 leader_election.cc:336] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2a9e035803f04da19fae285aa889e23d (127.31.48.190:36367): Network error: Client connection negotiation failed: client connection to 127.31.48.190:36367: connect: Connection refused (error 111)
I20250905 02:20:40.845613 1068 raft_consensus.cc:513] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:40.846550 632 raft_consensus.cc:2391] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 353be9287593454abec1e3bd6c5dd266 in current term 2: Already voted for candidate c39fe298fa9d4bbdb438c0799cce20a9 in this term.
I20250905 02:20:40.847378 1068 leader_election.cc:290] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [CANDIDATE]: Term 2 election: Requested vote from peers 2a9e035803f04da19fae285aa889e23d (127.31.48.190:36367), 353be9287593454abec1e3bd6c5dd266 (127.31.48.188:42595)
I20250905 02:20:40.848557 700 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "353be9287593454abec1e3bd6c5dd266"
I20250905 02:20:40.849133 700 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:20:40.849681 673 leader_election.cc:304] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 353be9287593454abec1e3bd6c5dd266; no voters: 2a9e035803f04da19fae285aa889e23d, c39fe298fa9d4bbdb438c0799cce20a9
W20250905 02:20:40.850926 610 leader_election.cc:336] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2a9e035803f04da19fae285aa889e23d (127.31.48.190:36367): Network error: Client connection negotiation failed: client connection to 127.31.48.190:36367: connect: Connection refused (error 111)
I20250905 02:20:40.853950 700 raft_consensus.cc:2466] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39fe298fa9d4bbdb438c0799cce20a9 in term 2.
I20250905 02:20:40.854359 1069 raft_consensus.cc:2747] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 02:20:40.854839 609 leader_election.cc:304] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 353be9287593454abec1e3bd6c5dd266, c39fe298fa9d4bbdb438c0799cce20a9; no voters: 2a9e035803f04da19fae285aa889e23d
I20250905 02:20:40.855474 1068 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:20:40.856652 1068 raft_consensus.cc:695] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 LEADER]: Becoming Leader. State: Replica: c39fe298fa9d4bbdb438c0799cce20a9, State: Running, Role: LEADER
I20250905 02:20:40.857421 1068 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } }
I20250905 02:20:40.860803 1074 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39fe298fa9d4bbdb438c0799cce20a9. Latest consensus state: current_term: 2 leader_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9e035803f04da19fae285aa889e23d" member_type: VOTER last_known_addr { host: "127.31.48.190" port: 36367 } } peers { permanent_uuid: "c39fe298fa9d4bbdb438c0799cce20a9" member_type: VOTER last_known_addr { host: "127.31.48.189" port: 38657 } } peers { permanent_uuid: "353be9287593454abec1e3bd6c5dd266" member_type: VOTER last_known_addr { host: "127.31.48.188" port: 42595 } } }
I20250905 02:20:40.861526 1074 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:40.862794 1076 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:40.867631 1076 catalog_manager.cc:671] Loaded metadata for table test-workload [id=149190314275419aaf8b48b28614d764]
I20250905 02:20:40.871943 1076 tablet_loader.cc:96] loaded metadata for tablet 77703afd391b469f89cb646f62c0e569 (table test-workload [id=149190314275419aaf8b48b28614d764])
I20250905 02:20:40.873196 1076 tablet_loader.cc:96] loaded metadata for tablet a7fd3bf8d7764d20ba07a1cbb94c394b (table test-workload [id=149190314275419aaf8b48b28614d764])
I20250905 02:20:40.874046 1076 tablet_loader.cc:96] loaded metadata for tablet d7c58f1a395b4581a7196f5a9d83ba15 (table test-workload [id=149190314275419aaf8b48b28614d764])
I20250905 02:20:40.874799 1076 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:40.877528 1076 catalog_manager.cc:1261] Loaded cluster ID: a2a053e889d248cb92a9a5a9af8cf3c6
I20250905 02:20:40.877699 1076 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:40.880750 1076 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:40.883509 1076 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: Loaded TSK: 0
I20250905 02:20:40.884299 1076 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:41.029625 767 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:41.030282 767 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:41.061754 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:20:41.085166 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:41.085938 31938 tablet_replica.cc:331] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d: stopping tablet replica
I20250905 02:20:41.086524 31938 raft_consensus.cc:2241] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:41.087136 31938 raft_consensus.cc:2270] T a7fd3bf8d7764d20ba07a1cbb94c394b P ca8c10e0f7b34e4d9c6288037010fa0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:41.106617 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:20:41.117385 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:20:41.136032 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:41.136802 31938 tablet_replica.cc:331] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a: stopping tablet replica
I20250905 02:20:41.137384 31938 raft_consensus.cc:2241] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:41.137959 31938 raft_consensus.cc:2270] T d7c58f1a395b4581a7196f5a9d83ba15 P b4ab8938afac48798dfcc396fa86bd7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:41.156155 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:20:41.165433 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:20:41.187691 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:20:41.188454 31938 tablet_replica.cc:331] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753: stopping tablet replica
I20250905 02:20:41.189047 31938 raft_consensus.cc:2241] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:20:41.189572 31938 raft_consensus.cc:2270] T 77703afd391b469f89cb646f62c0e569 P dab4364aa92f454194e281ca2a9bd753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:41.209345 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:20:41.218806 31938 master.cc:561] Master@127.31.48.189:38657 shutting down...
I20250905 02:20:41.237459 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:20:41.238365 31938 pending_rounds.cc:70] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: Trying to abort 1 pending ops.
I20250905 02:20:41.238593 31938 pending_rounds.cc:77] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: Aborting op as it isn't in flight: id { term: 2 index: 11 } timestamp: 7196831092158259200 op_type: NO_OP noop_request { }
I20250905 02:20:41.238966 31938 raft_consensus.cc:2887] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:20:41.239195 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:41.239578 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P c39fe298fa9d4bbdb438c0799cce20a9: stopping tablet replica
I20250905 02:20:41.259398 31938 master.cc:583] Master@127.31.48.189:38657 shutdown complete.
I20250905 02:20:41.274029 31938 master.cc:561] Master@127.31.48.188:42595 shutting down...
I20250905 02:20:41.290623 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:41.291376 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:41.291688 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 353be9287593454abec1e3bd6c5dd266: stopping tablet replica
I20250905 02:20:41.310468 31938 master.cc:583] Master@127.31.48.188:42595 shutdown complete.
[ OK ] AutoRebalancerTest.NextLeaderResumesAutoRebalancing (4684 ms)
[ RUN ] AutoRebalancerTest.MovesScheduledIfAddTserver
I20250905 02:20:41.342164 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:41501
I20250905 02:20:41.343204 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:41.348277 1078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.348971 1079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.349581 1081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:41.350219 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:41.350893 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:41.351087 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:41.351230 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038841351214 us; error 0 us; skew 500 ppm
I20250905 02:20:41.351702 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:41.354007 31938 webserver.cc:480] Webserver started at http://127.31.48.190:33527/ using document root <none> and password file <none>
I20250905 02:20:41.354468 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:41.354642 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:41.354887 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:41.356097 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "4b6afeb8723446539a1b7d21f6920de5"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.360416 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:20:41.363639 1086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.364408 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250905 02:20:41.364686 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "4b6afeb8723446539a1b7d21f6920de5"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.364940 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:41.377550 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:41.378727 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:41.415660 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:41501
I20250905 02:20:41.415756 1137 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:41501 every 8 connection(s)
I20250905 02:20:41.419637 1138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:41.429318 1138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5: Bootstrap starting.
I20250905 02:20:41.433790 1138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:41.437850 1138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5: No bootstrap required, opened a new log
I20250905 02:20:41.439823 1138 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6afeb8723446539a1b7d21f6920de5" member_type: VOTER }
I20250905 02:20:41.440213 1138 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:41.440523 1138 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6afeb8723446539a1b7d21f6920de5, State: Initialized, Role: FOLLOWER
I20250905 02:20:41.441066 1138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6afeb8723446539a1b7d21f6920de5" member_type: VOTER }
I20250905 02:20:41.441497 1138 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:20:41.441759 1138 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:20:41.442065 1138 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:41.446135 1138 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6afeb8723446539a1b7d21f6920de5" member_type: VOTER }
I20250905 02:20:41.446602 1138 leader_election.cc:304] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6afeb8723446539a1b7d21f6920de5; no voters:
I20250905 02:20:41.447717 1138 leader_election.cc:290] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:20:41.448118 1141 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:41.449410 1141 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6afeb8723446539a1b7d21f6920de5, State: Running, Role: LEADER
I20250905 02:20:41.450058 1141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6afeb8723446539a1b7d21f6920de5" member_type: VOTER }
I20250905 02:20:41.450564 1138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:20:41.453554 1143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6afeb8723446539a1b7d21f6920de5. Latest consensus state: current_term: 1 leader_uuid: "4b6afeb8723446539a1b7d21f6920de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6afeb8723446539a1b7d21f6920de5" member_type: VOTER } }
I20250905 02:20:41.453527 1142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6afeb8723446539a1b7d21f6920de5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6afeb8723446539a1b7d21f6920de5" member_type: VOTER } }
I20250905 02:20:41.455228 1143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:41.455255 1142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [sys.catalog]: This master's current role is: LEADER
I20250905 02:20:41.457593 1149 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:20:41.464306 1149 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:20:41.465428 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:20:41.472502 1149 catalog_manager.cc:1349] Generated new cluster ID: 6db73d3baf4740c9920961644f565102
I20250905 02:20:41.472810 1149 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:20:41.494832 1149 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:20:41.496143 1149 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:20:41.510345 1149 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5: Generated new TSK 0
I20250905 02:20:41.510985 1149 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:20:41.532052 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:41.538117 1159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.539004 1160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.540570 1162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:41.541276 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:41.542061 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:41.542263 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:41.542389 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038841542378 us; error 0 us; skew 500 ppm
I20250905 02:20:41.542833 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:41.545089 31938 webserver.cc:480] Webserver started at http://127.31.48.129:35123/ using document root <none> and password file <none>
I20250905 02:20:41.545542 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:41.545692 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:41.545902 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:41.546880 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.551020 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:20:41.554376 1167 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.555148 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:20:41.555382 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.555600 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:41.571435 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:41.572631 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:41.574031 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:41.576332 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:41.576633 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.576949 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:41.577207 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.616649 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:33195
I20250905 02:20:41.616745 1229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:33195 every 8 connection(s)
I20250905 02:20:41.622480 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:41.629482 1234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.631700 1235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:41.633702 31938 server_base.cc:1047] running on GCE node
W20250905 02:20:41.635247 1237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:41.636272 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:41.636577 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:41.636786 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038841636770 us; error 0 us; skew 500 ppm
I20250905 02:20:41.637409 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:41.637715 1230 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41501
I20250905 02:20:41.638106 1230 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:41.638850 1230 heartbeater.cc:507] Master 127.31.48.190:41501 requested a full tablet report, sending...
I20250905 02:20:41.640720 31938 webserver.cc:480] Webserver started at http://127.31.48.130:42677/ using document root <none> and password file <none>
I20250905 02:20:41.641238 1103 ts_manager.cc:194] Registered new tserver with Master: 6128b6e3cf474bd3bc4eadbebcb2ecc0 (127.31.48.129:33195)
I20250905 02:20:41.641430 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:41.641717 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:41.642053 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:41.643484 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "689e9b0add644fe5a5dc813eaf521884"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.643771 1103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46892
I20250905 02:20:41.648250 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20250905 02:20:41.651422 1242 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.652245 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:20:41.652539 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "689e9b0add644fe5a5dc813eaf521884"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.652806 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:41.660669 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:41.662003 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:41.663297 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:41.665395 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:41.665594 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.665825 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:41.665961 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.703493 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:33057
I20250905 02:20:41.703585 1304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:33057 every 8 connection(s)
I20250905 02:20:41.707893 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:41.714833 1308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.716774 1309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:41.718467 1311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:41.721724 1305 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41501
I20250905 02:20:41.721776 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:41.722325 1305 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:41.723021 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:41.723296 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:41.723238 1305 heartbeater.cc:507] Master 127.31.48.190:41501 requested a full tablet report, sending...
I20250905 02:20:41.723634 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038841723613 us; error 0 us; skew 500 ppm
I20250905 02:20:41.724363 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:41.725494 1103 ts_manager.cc:194] Registered new tserver with Master: 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130:33057)
I20250905 02:20:41.727136 1103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46898
I20250905 02:20:41.729516 31938 webserver.cc:480] Webserver started at http://127.31.48.131:40341/ using document root <none> and password file <none>
I20250905 02:20:41.729992 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:41.730165 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:41.730370 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:41.731472 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "cf73963d1c1b462da8d3c7a782823223"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.736202 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20250905 02:20:41.739497 1316 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.740298 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:20:41.740644 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "cf73963d1c1b462da8d3c7a782823223"
format_stamp: "Formatted at 2025-09-05 02:20:41 on dist-test-slave-q32q"
I20250905 02:20:41.740931 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:41.763821 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:41.765266 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:41.766722 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:41.768977 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:41.769171 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.769384 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:41.769532 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:41.809689 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:41115
I20250905 02:20:41.809775 1378 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:41115 every 8 connection(s)
I20250905 02:20:41.823519 1379 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41501
I20250905 02:20:41.823923 1379 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:41.824676 1379 heartbeater.cc:507] Master 127.31.48.190:41501 requested a full tablet report, sending...
I20250905 02:20:41.826570 1103 ts_manager.cc:194] Registered new tserver with Master: cf73963d1c1b462da8d3c7a782823223 (127.31.48.131:41115)
I20250905 02:20:41.827440 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01463341s
I20250905 02:20:41.828176 1103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46902
I20250905 02:20:42.646543 1230 heartbeater.cc:499] Master 127.31.48.190:41501 was elected leader, sending a full tablet report...
I20250905 02:20:42.729981 1305 heartbeater.cc:499] Master 127.31.48.190:41501 was elected leader, sending a full tablet report...
I20250905 02:20:42.830850 1379 heartbeater.cc:499] Master 127.31.48.190:41501 was elected leader, sending a full tablet report...
I20250905 02:20:42.860844 31938 test_util.cc:276] Using random seed: 2081805589
I20250905 02:20:42.883674 1103 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46908:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:20:42.886234 1103 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:20:42.941501 1270 tablet_service.cc:1468] Processing CreateTablet for tablet 285c5f7d1c69461d961ce6da7bffc89d (DEFAULT_TABLE table=test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250905 02:20:42.942291 1344 tablet_service.cc:1468] Processing CreateTablet for tablet 285c5f7d1c69461d961ce6da7bffc89d (DEFAULT_TABLE table=test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250905 02:20:42.943176 1270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285c5f7d1c69461d961ce6da7bffc89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:42.943351 1343 tablet_service.cc:1468] Processing CreateTablet for tablet 71ceb0fd1848477d8c0ba33e6de5fb4d (DEFAULT_TABLE table=test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250905 02:20:42.943714 1344 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285c5f7d1c69461d961ce6da7bffc89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:42.941578 1269 tablet_service.cc:1468] Processing CreateTablet for tablet 71ceb0fd1848477d8c0ba33e6de5fb4d (DEFAULT_TABLE table=test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250905 02:20:42.944555 1343 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ceb0fd1848477d8c0ba33e6de5fb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:42.946467 1269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ceb0fd1848477d8c0ba33e6de5fb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:42.955711 1195 tablet_service.cc:1468] Processing CreateTablet for tablet 285c5f7d1c69461d961ce6da7bffc89d (DEFAULT_TABLE table=test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20250905 02:20:42.957144 1195 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285c5f7d1c69461d961ce6da7bffc89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:42.964521 1194 tablet_service.cc:1468] Processing CreateTablet for tablet 71ceb0fd1848477d8c0ba33e6de5fb4d (DEFAULT_TABLE table=test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20250905 02:20:42.965584 1194 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71ceb0fd1848477d8c0ba33e6de5fb4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:42.968271 1399 tablet_bootstrap.cc:492] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223: Bootstrap starting.
I20250905 02:20:42.977164 1399 tablet_bootstrap.cc:654] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:42.983788 1400 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: Bootstrap starting.
I20250905 02:20:42.989652 1399 tablet_bootstrap.cc:492] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223: No bootstrap required, opened a new log
I20250905 02:20:42.990355 1399 ts_tablet_manager.cc:1397] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20250905 02:20:42.992961 1400 tablet_bootstrap.cc:654] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:42.993427 1399 raft_consensus.cc:357] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:42.994422 1399 raft_consensus.cc:383] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:42.994642 1401 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Bootstrap starting.
I20250905 02:20:42.994766 1399 raft_consensus.cc:738] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf73963d1c1b462da8d3c7a782823223, State: Initialized, Role: FOLLOWER
I20250905 02:20:42.995877 1399 consensus_queue.cc:260] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.001395 1400 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: No bootstrap required, opened a new log
I20250905 02:20:43.002010 1400 ts_tablet_manager.cc:1397] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20250905 02:20:43.001370 1399 ts_tablet_manager.cc:1428] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20250905 02:20:43.003201 1401 tablet_bootstrap.cc:654] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:43.003342 1399 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223: Bootstrap starting.
I20250905 02:20:43.004705 1400 raft_consensus.cc:357] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.005508 1400 raft_consensus.cc:383] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:43.005853 1400 raft_consensus.cc:738] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689e9b0add644fe5a5dc813eaf521884, State: Initialized, Role: FOLLOWER
I20250905 02:20:43.020121 1400 consensus_queue.cc:260] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.020741 1399 tablet_bootstrap.cc:654] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:43.022420 1401 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: No bootstrap required, opened a new log
I20250905 02:20:43.022989 1401 ts_tablet_manager.cc:1397] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.008s
I20250905 02:20:43.025980 1400 ts_tablet_manager.cc:1428] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: Time spent starting tablet: real 0.024s user 0.006s sys 0.000s
I20250905 02:20:43.026207 1401 raft_consensus.cc:357] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.026985 1400 tablet_bootstrap.cc:492] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884: Bootstrap starting.
I20250905 02:20:43.027201 1401 raft_consensus.cc:383] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:43.027750 1401 raft_consensus.cc:738] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6128b6e3cf474bd3bc4eadbebcb2ecc0, State: Initialized, Role: FOLLOWER
I20250905 02:20:43.028766 1401 consensus_queue.cc:260] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.033669 1400 tablet_bootstrap.cc:654] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:43.036232 1401 ts_tablet_manager.cc:1428] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20250905 02:20:43.038286 1401 tablet_bootstrap.cc:492] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Bootstrap starting.
I20250905 02:20:43.036908 1399 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223: No bootstrap required, opened a new log
I20250905 02:20:43.043267 1399 ts_tablet_manager.cc:1397] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.013s
I20250905 02:20:43.045325 1401 tablet_bootstrap.cc:654] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Neither blocks nor log segments found. Creating new log.
I20250905 02:20:43.045352 1400 tablet_bootstrap.cc:492] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884: No bootstrap required, opened a new log
I20250905 02:20:43.046466 1400 ts_tablet_manager.cc:1397] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20250905 02:20:43.046202 1399 raft_consensus.cc:357] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.047022 1399 raft_consensus.cc:383] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:43.047392 1399 raft_consensus.cc:738] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf73963d1c1b462da8d3c7a782823223, State: Initialized, Role: FOLLOWER
I20250905 02:20:43.048206 1399 consensus_queue.cc:260] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.049151 1408 raft_consensus.cc:491] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:43.049832 1408 raft_consensus.cc:513] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.052289 1400 raft_consensus.cc:357] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.053098 1400 raft_consensus.cc:383] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:43.053429 1400 raft_consensus.cc:738] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 689e9b0add644fe5a5dc813eaf521884, State: Initialized, Role: FOLLOWER
I20250905 02:20:43.054029 1399 ts_tablet_manager.cc:1428] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250905 02:20:43.054250 1400 consensus_queue.cc:260] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.056470 1408 leader_election.cc:290] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130:33057), cf73963d1c1b462da8d3c7a782823223 (127.31.48.131:41115)
I20250905 02:20:43.063753 1400 ts_tablet_manager.cc:1428] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20250905 02:20:43.064481 1401 tablet_bootstrap.cc:492] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: No bootstrap required, opened a new log
I20250905 02:20:43.065122 1401 ts_tablet_manager.cc:1397] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20250905 02:20:43.068334 1401 raft_consensus.cc:357] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.073285 1401 raft_consensus.cc:383] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:20:43.073683 1401 raft_consensus.cc:738] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6128b6e3cf474bd3bc4eadbebcb2ecc0, State: Initialized, Role: FOLLOWER
I20250905 02:20:43.074539 1401 consensus_queue.cc:260] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.076953 1401 ts_tablet_manager.cc:1428] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Time spent starting tablet: real 0.011s user 0.000s sys 0.008s
I20250905 02:20:43.080269 1280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "285c5f7d1c69461d961ce6da7bffc89d" candidate_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689e9b0add644fe5a5dc813eaf521884" is_pre_election: true
I20250905 02:20:43.081101 1280 raft_consensus.cc:2466] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6128b6e3cf474bd3bc4eadbebcb2ecc0 in term 0.
I20250905 02:20:43.082361 1169 leader_election.cc:304] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6128b6e3cf474bd3bc4eadbebcb2ecc0, 689e9b0add644fe5a5dc813eaf521884; no voters:
I20250905 02:20:43.083144 1408 raft_consensus.cc:2802] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:43.083513 1408 raft_consensus.cc:491] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:43.083839 1408 raft_consensus.cc:3058] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:43.083583 1354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "285c5f7d1c69461d961ce6da7bffc89d" candidate_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf73963d1c1b462da8d3c7a782823223" is_pre_election: true
I20250905 02:20:43.084398 1354 raft_consensus.cc:2466] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6128b6e3cf474bd3bc4eadbebcb2ecc0 in term 0.
I20250905 02:20:43.095237 1403 raft_consensus.cc:491] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:20:43.095679 1403 raft_consensus.cc:513] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.097609 1403 leader_election.cc:290] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6128b6e3cf474bd3bc4eadbebcb2ecc0 (127.31.48.129:33195), 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130:33057)
I20250905 02:20:43.107052 1205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ceb0fd1848477d8c0ba33e6de5fb4d" candidate_uuid: "cf73963d1c1b462da8d3c7a782823223" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" is_pre_election: true
I20250905 02:20:43.107673 1205 raft_consensus.cc:2466] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf73963d1c1b462da8d3c7a782823223 in term 0.
I20250905 02:20:43.108418 1280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ceb0fd1848477d8c0ba33e6de5fb4d" candidate_uuid: "cf73963d1c1b462da8d3c7a782823223" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689e9b0add644fe5a5dc813eaf521884" is_pre_election: true
I20250905 02:20:43.108942 1280 raft_consensus.cc:2466] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf73963d1c1b462da8d3c7a782823223 in term 0.
I20250905 02:20:43.108844 1318 leader_election.cc:304] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6128b6e3cf474bd3bc4eadbebcb2ecc0, cf73963d1c1b462da8d3c7a782823223; no voters:
I20250905 02:20:43.109663 1403 raft_consensus.cc:2802] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:20:43.110015 1403 raft_consensus.cc:491] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:20:43.110324 1403 raft_consensus.cc:3058] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:43.121279 1403 raft_consensus.cc:513] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.121326 1408 raft_consensus.cc:513] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.123881 1408 leader_election.cc:290] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [CANDIDATE]: Term 1 election: Requested vote from peers 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130:33057), cf73963d1c1b462da8d3c7a782823223 (127.31.48.131:41115)
I20250905 02:20:43.125322 1403 leader_election.cc:290] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [CANDIDATE]: Term 1 election: Requested vote from peers 6128b6e3cf474bd3bc4eadbebcb2ecc0 (127.31.48.129:33195), 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130:33057)
I20250905 02:20:43.126190 1205 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ceb0fd1848477d8c0ba33e6de5fb4d" candidate_uuid: "cf73963d1c1b462da8d3c7a782823223" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0"
I20250905 02:20:43.126791 1205 raft_consensus.cc:3058] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:43.124801 1280 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "285c5f7d1c69461d961ce6da7bffc89d" candidate_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689e9b0add644fe5a5dc813eaf521884"
I20250905 02:20:43.127229 1279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "71ceb0fd1848477d8c0ba33e6de5fb4d" candidate_uuid: "cf73963d1c1b462da8d3c7a782823223" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "689e9b0add644fe5a5dc813eaf521884"
I20250905 02:20:43.127833 1280 raft_consensus.cc:3058] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:43.128203 1279 raft_consensus.cc:3058] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:43.125946 1354 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "285c5f7d1c69461d961ce6da7bffc89d" candidate_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf73963d1c1b462da8d3c7a782823223"
I20250905 02:20:43.135097 1354 raft_consensus.cc:3058] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:20:43.227217 1354 raft_consensus.cc:2466] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6128b6e3cf474bd3bc4eadbebcb2ecc0 in term 1.
I20250905 02:20:43.227223 1205 raft_consensus.cc:2466] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf73963d1c1b462da8d3c7a782823223 in term 1.
I20250905 02:20:43.227222 1279 raft_consensus.cc:2466] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf73963d1c1b462da8d3c7a782823223 in term 1.
I20250905 02:20:43.227509 1280 raft_consensus.cc:2466] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6128b6e3cf474bd3bc4eadbebcb2ecc0 in term 1.
I20250905 02:20:43.229307 1318 leader_election.cc:304] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6128b6e3cf474bd3bc4eadbebcb2ecc0, cf73963d1c1b462da8d3c7a782823223; no voters:
I20250905 02:20:43.229203 1171 leader_election.cc:304] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6128b6e3cf474bd3bc4eadbebcb2ecc0, cf73963d1c1b462da8d3c7a782823223; no voters:
I20250905 02:20:43.230108 1403 raft_consensus.cc:2802] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:43.230324 1408 raft_consensus.cc:2802] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:20:43.232501 1408 raft_consensus.cc:695] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 LEADER]: Becoming Leader. State: Replica: 6128b6e3cf474bd3bc4eadbebcb2ecc0, State: Running, Role: LEADER
I20250905 02:20:43.233423 1408 consensus_queue.cc:237] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } }
I20250905 02:20:43.234185 1403 raft_consensus.cc:695] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 1 LEADER]: Becoming Leader. State: Replica: cf73963d1c1b462da8d3c7a782823223, State: Running, Role: LEADER
I20250905 02:20:43.235420 1403 consensus_queue.cc:237] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } }
I20250905 02:20:43.240985 1103 catalog_manager.cc:5582] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6128b6e3cf474bd3bc4eadbebcb2ecc0 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:43.242539 1102 catalog_manager.cc:5582] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf73963d1c1b462da8d3c7a782823223 (127.31.48.131). New cstate: current_term: 1 leader_uuid: "cf73963d1c1b462da8d3c7a782823223" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } health_report { overall_health: HEALTHY } } }
I20250905 02:20:43.465272 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:43.467257 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:43.467631 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:43.751173 1419 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:43.764575 1424 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:43.771863 1426 consensus_queue.cc:1035] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:43.783715 1426 consensus_queue.cc:1035] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:20:45.468498 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:45.470472 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:45.470839 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:47.471707 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:47.473601 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:47.473923 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:49.474799 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:49.476759 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:49.477109 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:51.478335 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:51.480579 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:51.480865 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:53.481709 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:53.483520 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:53.483819 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:54.356993 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:20:54.364094 1437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:54.364384 1438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:20:54.366132 1440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:20:54.367300 31938 server_base.cc:1047] running on GCE node
I20250905 02:20:54.368196 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:20:54.368440 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:20:54.368587 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038854368574 us; error 0 us; skew 500 ppm
I20250905 02:20:54.369040 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:20:54.371500 31938 webserver.cc:480] Webserver started at http://127.31.48.132:44781/ using document root <none> and password file <none>
I20250905 02:20:54.371949 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:20:54.372128 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:20:54.372419 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:20:54.373754 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "5c99bb21b3e34b089a26e20bd6199c80"
format_stamp: "Formatted at 2025-09-05 02:20:54 on dist-test-slave-q32q"
I20250905 02:20:54.378479 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250905 02:20:54.381767 1445 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:54.382530 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:20:54.382802 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "5c99bb21b3e34b089a26e20bd6199c80"
format_stamp: "Formatted at 2025-09-05 02:20:54 on dist-test-slave-q32q"
I20250905 02:20:54.383082 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:20:54.403692 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:20:54.405047 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:20:54.406531 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:20:54.408939 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:20:54.409158 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:54.409389 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:20:54.409554 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:20:54.448942 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:44319
I20250905 02:20:54.449034 1507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:44319 every 8 connection(s)
I20250905 02:20:54.468710 1508 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41501
I20250905 02:20:54.469127 1508 heartbeater.cc:461] Registering TS with master...
I20250905 02:20:54.469875 1508 heartbeater.cc:507] Master 127.31.48.190:41501 requested a full tablet report, sending...
I20250905 02:20:54.471691 1103 ts_manager.cc:194] Registered new tserver with Master: 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132:44319)
I20250905 02:20:54.473126 1103 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38096
I20250905 02:20:54.523828 1205 consensus_queue.cc:237] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } }
I20250905 02:20:54.533169 1354 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:54.533414 1279 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:20:54.534633 1434 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:20:54.536077 1434 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:20:54.544225 1169 consensus_peers.cc:489] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 -> Peer 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132:44319): Couldn't send request to peer 5c99bb21b3e34b089a26e20bd6199c80. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 285c5f7d1c69461d961ce6da7bffc89d. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:20:54.545258 1516 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } } }
I20250905 02:20:54.547077 1354 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } } }
I20250905 02:20:54.547472 1279 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } } }
I20250905 02:20:54.557328 1103 catalog_manager.cc:5582] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:20:55.018051 1526 ts_tablet_manager.cc:927] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: Initiating tablet copy from peer 6128b6e3cf474bd3bc4eadbebcb2ecc0 (127.31.48.129:33195)
I20250905 02:20:55.019683 1526 tablet_copy_client.cc:323] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.129:33195
I20250905 02:20:55.031420 1215 tablet_copy_service.cc:140] P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Received BeginTabletCopySession request for tablet 285c5f7d1c69461d961ce6da7bffc89d from peer 5c99bb21b3e34b089a26e20bd6199c80 ({username='slave'} at 127.0.0.1:44234)
I20250905 02:20:55.031951 1215 tablet_copy_service.cc:161] P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Beginning new tablet copy session on tablet 285c5f7d1c69461d961ce6da7bffc89d from peer 5c99bb21b3e34b089a26e20bd6199c80 at {username='slave'} at 127.0.0.1:44234: session id = 5c99bb21b3e34b089a26e20bd6199c80-285c5f7d1c69461d961ce6da7bffc89d
I20250905 02:20:55.039124 1215 tablet_copy_source_session.cc:215] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:20:55.042280 1526 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 285c5f7d1c69461d961ce6da7bffc89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:20:55.054983 1526 tablet_copy_client.cc:806] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: tablet copy: Starting download of 0 data blocks...
I20250905 02:20:55.055548 1526 tablet_copy_client.cc:670] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: tablet copy: Starting download of 1 WAL segments...
I20250905 02:20:55.059763 1526 tablet_copy_client.cc:538] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:20:55.067620 1526 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: Bootstrap starting.
I20250905 02:20:55.085601 1526 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:20:55.086539 1526 tablet_bootstrap.cc:492] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: Bootstrap complete.
I20250905 02:20:55.087127 1526 ts_tablet_manager.cc:1397] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.005s
I20250905 02:20:55.089659 1526 raft_consensus.cc:357] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } }
I20250905 02:20:55.090246 1526 raft_consensus.cc:738] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5c99bb21b3e34b089a26e20bd6199c80, State: Initialized, Role: LEARNER
I20250905 02:20:55.090890 1526 consensus_queue.cc:260] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: true } }
I20250905 02:20:55.093048 1508 heartbeater.cc:499] Master 127.31.48.190:41501 was elected leader, sending a full tablet report...
I20250905 02:20:55.093597 1526 ts_tablet_manager.cc:1428] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250905 02:20:55.095304 1215 tablet_copy_service.cc:342] P 6128b6e3cf474bd3bc4eadbebcb2ecc0: Request end of tablet copy session 5c99bb21b3e34b089a26e20bd6199c80-285c5f7d1c69461d961ce6da7bffc89d received from {username='slave'} at 127.0.0.1:44234
I20250905 02:20:55.095693 1215 tablet_copy_service.cc:434] P 6128b6e3cf474bd3bc4eadbebcb2ecc0: ending tablet copy session 5c99bb21b3e34b089a26e20bd6199c80-285c5f7d1c69461d961ce6da7bffc89d on tablet 285c5f7d1c69461d961ce6da7bffc89d with peer 5c99bb21b3e34b089a26e20bd6199c80
I20250905 02:20:55.484701 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:55.486671 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:55.487059 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:55.568506 1483 raft_consensus.cc:1215] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 02:20:56.021617 1532 raft_consensus.cc:1062] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: attempting to promote NON_VOTER 5c99bb21b3e34b089a26e20bd6199c80 to VOTER
I20250905 02:20:56.023759 1532 consensus_queue.cc:237] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } }
I20250905 02:20:56.029569 1483 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 LEARNER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:20:56.029773 1354 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:20:56.031245 1279 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:20:56.031160 1531 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:20:56.032554 1532 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:20:56.033805 1515 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:20:56.046002 1515 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.047494 1354 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.048204 1279 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.057346 1483 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.059574 1101 catalog_manager.cc:5582] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 reported cstate change: config changed from index 2 to 3, 5c99bb21b3e34b089a26e20bd6199c80 (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "689e9b0add644fe5a5dc813eaf521884" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 33057 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 02:20:56.079998 1204 consensus_queue.cc:237] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } }
I20250905 02:20:56.084964 1354 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:20:56.084990 1483 raft_consensus.cc:1273] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 FOLLOWER]: Refusing update from remote peer 6128b6e3cf474bd3bc4eadbebcb2ecc0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:20:56.086454 1532 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:20:56.087333 1531 consensus_queue.cc:1035] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:20:56.093343 1531 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.094936 1483 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.098774 1354 raft_consensus.cc:2953] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } } }
I20250905 02:20:56.101107 1088 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 285c5f7d1c69461d961ce6da7bffc89d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:20:56.105214 1100 catalog_manager.cc:5582] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 reported cstate change: config changed from index 3 to 4, VOTER 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130) evicted. New cstate: current_term: 1 leader_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cf73963d1c1b462da8d3c7a782823223" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6128b6e3cf474bd3bc4eadbebcb2ecc0" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c99bb21b3e34b089a26e20bd6199c80" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 44319 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 02:20:56.116732 1270 tablet_service.cc:1515] Processing DeleteTablet for tablet 285c5f7d1c69461d961ce6da7bffc89d with delete_type TABLET_DATA_TOMBSTONED (TS 689e9b0add644fe5a5dc813eaf521884 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44672
I20250905 02:20:56.118932 1542 tablet_replica.cc:331] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: stopping tablet replica
I20250905 02:20:56.119814 1542 raft_consensus.cc:2241] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:20:56.120493 1542 raft_consensus.cc:2270] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:20:56.122869 1542 ts_tablet_manager.cc:1905] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:20:56.133055 1542 ts_tablet_manager.cc:1918] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 02:20:56.133424 1542 log.cc:1199] T 285c5f7d1c69461d961ce6da7bffc89d P 689e9b0add644fe5a5dc813eaf521884: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.MovesScheduledIfAddTserver.1757038820745333-31938-0/minicluster-data/ts-1-root/wals/285c5f7d1c69461d961ce6da7bffc89d
I20250905 02:20:56.134716 1088 catalog_manager.cc:4928] TS 689e9b0add644fe5a5dc813eaf521884 (127.31.48.130:33057): tablet 285c5f7d1c69461d961ce6da7bffc89d (table test-workload [id=b01cd5fb4e64452c89aabbdaefacd484]) successfully deleted
I20250905 02:20:57.488013 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:57.489952 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:57.490325 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:20:59.491233 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:20:59.493165 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:20:59.493476 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:01.494537 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:01.497052 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:01.497423 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:03.498504 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:03.500516 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:03.500876 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:05.501811 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:05.503747 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:05.504091 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:07.505048 1156 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:07.506927 1156 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:07.507262 1156 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:08.165396 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:21:08.193253 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:08.193940 31938 tablet_replica.cc:331] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: stopping tablet replica
I20250905 02:21:08.194603 31938 raft_consensus.cc:2241] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:08.195143 31938 raft_consensus.cc:2270] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.197369 31938 tablet_replica.cc:331] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0: stopping tablet replica
I20250905 02:21:08.197819 31938 raft_consensus.cc:2241] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:08.198685 31938 raft_consensus.cc:2270] T 285c5f7d1c69461d961ce6da7bffc89d P 6128b6e3cf474bd3bc4eadbebcb2ecc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.222234 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:21:08.235039 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:21:08.258185 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:08.258788 31938 tablet_replica.cc:331] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884: stopping tablet replica
I20250905 02:21:08.259341 31938 raft_consensus.cc:2241] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:08.259816 31938 raft_consensus.cc:2270] T 71ceb0fd1848477d8c0ba33e6de5fb4d P 689e9b0add644fe5a5dc813eaf521884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.280484 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:21:08.291620 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
W20250905 02:21:08.314322 1318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33195: connect: Connection refused (error 111) [suppressed 17 similar messages]
I20250905 02:21:08.315757 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:08.316483 31938 tablet_replica.cc:331] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223: stopping tablet replica
I20250905 02:21:08.317224 31938 raft_consensus.cc:2241] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:08.317912 31938 raft_consensus.cc:2270] T 285c5f7d1c69461d961ce6da7bffc89d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.319937 31938 tablet_replica.cc:331] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223: stopping tablet replica
I20250905 02:21:08.320482 31938 raft_consensus.cc:2241] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:08.321523 31938 raft_consensus.cc:2270] T 71ceb0fd1848477d8c0ba33e6de5fb4d P cf73963d1c1b462da8d3c7a782823223 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.341958 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:21:08.353518 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:21:08.376283 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:08.376960 31938 tablet_replica.cc:331] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80: stopping tablet replica
I20250905 02:21:08.377506 31938 raft_consensus.cc:2241] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:08.378100 31938 raft_consensus.cc:2270] T 285c5f7d1c69461d961ce6da7bffc89d P 5c99bb21b3e34b089a26e20bd6199c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.397119 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:21:08.407063 31938 master.cc:561] Master@127.31.48.190:41501 shutting down...
I20250905 02:21:08.425801 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:08.426409 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:08.426710 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 4b6afeb8723446539a1b7d21f6920de5: stopping tablet replica
I20250905 02:21:08.445736 31938 master.cc:583] Master@127.31.48.190:41501 shutdown complete.
[ OK ] AutoRebalancerTest.MovesScheduledIfAddTserver (27140 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTservers
I20250905 02:21:08.482481 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:37413
I20250905 02:21:08.483628 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:08.489995 1550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:08.490020 1551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:08.490448 1553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:08.491532 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:08.492399 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:08.492606 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:08.492763 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038868492746 us; error 0 us; skew 500 ppm
I20250905 02:21:08.493248 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:08.495543 31938 webserver.cc:480] Webserver started at http://127.31.48.190:42687/ using document root <none> and password file <none>
I20250905 02:21:08.496006 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:08.496192 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:08.496491 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:08.497617 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "dd8d26fc911045709f545375b46ea1ff"
format_stamp: "Formatted at 2025-09-05 02:21:08 on dist-test-slave-q32q"
I20250905 02:21:08.502123 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250905 02:21:08.505481 1558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:08.506232 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:21:08.506500 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "dd8d26fc911045709f545375b46ea1ff"
format_stamp: "Formatted at 2025-09-05 02:21:08 on dist-test-slave-q32q"
I20250905 02:21:08.506775 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTservers.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:08.529706 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:08.530870 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:08.567353 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:37413
I20250905 02:21:08.567464 1609 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:37413 every 8 connection(s)
I20250905 02:21:08.571281 1610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:08.581491 1610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff: Bootstrap starting.
I20250905 02:21:08.585906 1610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:08.589938 1610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff: No bootstrap required, opened a new log
I20250905 02:21:08.592025 1610 raft_consensus.cc:357] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8d26fc911045709f545375b46ea1ff" member_type: VOTER }
I20250905 02:21:08.592496 1610 raft_consensus.cc:383] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:08.592734 1610 raft_consensus.cc:738] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8d26fc911045709f545375b46ea1ff, State: Initialized, Role: FOLLOWER
I20250905 02:21:08.593277 1610 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8d26fc911045709f545375b46ea1ff" member_type: VOTER }
I20250905 02:21:08.593739 1610 raft_consensus.cc:397] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:08.593952 1610 raft_consensus.cc:491] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:08.594238 1610 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:08.598465 1610 raft_consensus.cc:513] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8d26fc911045709f545375b46ea1ff" member_type: VOTER }
I20250905 02:21:08.598945 1610 leader_election.cc:304] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8d26fc911045709f545375b46ea1ff; no voters:
I20250905 02:21:08.599997 1610 leader_election.cc:290] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:08.600325 1613 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:08.601680 1613 raft_consensus.cc:695] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 1 LEADER]: Becoming Leader. State: Replica: dd8d26fc911045709f545375b46ea1ff, State: Running, Role: LEADER
I20250905 02:21:08.602283 1613 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8d26fc911045709f545375b46ea1ff" member_type: VOTER }
I20250905 02:21:08.602895 1610 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:08.606782 1614 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8d26fc911045709f545375b46ea1ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8d26fc911045709f545375b46ea1ff" member_type: VOTER } }
I20250905 02:21:08.608129 1614 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:08.607542 1615 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd8d26fc911045709f545375b46ea1ff. Latest consensus state: current_term: 1 leader_uuid: "dd8d26fc911045709f545375b46ea1ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8d26fc911045709f545375b46ea1ff" member_type: VOTER } }
I20250905 02:21:08.609290 1615 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:08.610167 1621 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:08.615123 1621 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:08.617489 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:21:08.624491 1621 catalog_manager.cc:1349] Generated new cluster ID: ce27a89222d14a63a66a57b870db5e37
I20250905 02:21:08.624821 1621 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:08.645948 1621 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:21:08.647212 1621 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:08.660800 1621 catalog_manager.cc:5955] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff: Generated new TSK 0
I20250905 02:21:08.661463 1621 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:21:08.682889 31938 internal_mini_cluster.cc:371] 0 TS(s) registered with all masters after 0.000190445s
I20250905 02:21:20.777338 31938 master.cc:561] Master@127.31.48.190:37413 shutting down...
I20250905 02:21:20.800884 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:20.801699 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:20.802043 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P dd8d26fc911045709f545375b46ea1ff: stopping tablet replica
I20250905 02:21:20.821041 31938 master.cc:583] Master@127.31.48.190:37413 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTservers (12373 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfNoTablets
I20250905 02:21:20.856105 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:37101
I20250905 02:21:20.857494 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:20.862685 1633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:20.863395 1634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:20.866992 31938 server_base.cc:1047] running on GCE node
W20250905 02:21:20.866945 1636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:20.867981 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:20.868203 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:20.868347 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038880868334 us; error 0 us; skew 500 ppm
I20250905 02:21:20.868858 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:20.871342 31938 webserver.cc:480] Webserver started at http://127.31.48.190:43649/ using document root <none> and password file <none>
I20250905 02:21:20.871798 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:20.871965 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:20.872222 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:20.873430 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "2679e30a20f242e39dcbfbdd22f8db46"
format_stamp: "Formatted at 2025-09-05 02:21:20 on dist-test-slave-q32q"
I20250905 02:21:20.877902 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:21:20.881274 1641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:20.881964 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:20.882227 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "2679e30a20f242e39dcbfbdd22f8db46"
format_stamp: "Formatted at 2025-09-05 02:21:20 on dist-test-slave-q32q"
I20250905 02:21:20.882475 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:20.901885 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:20.903059 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:20.940757 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:37101
I20250905 02:21:20.940826 1692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:37101 every 8 connection(s)
I20250905 02:21:20.945327 1693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:20.956826 1693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46: Bootstrap starting.
I20250905 02:21:20.961450 1693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:20.965569 1693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46: No bootstrap required, opened a new log
I20250905 02:21:20.967645 1693 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2679e30a20f242e39dcbfbdd22f8db46" member_type: VOTER }
I20250905 02:21:20.968067 1693 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:20.968264 1693 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2679e30a20f242e39dcbfbdd22f8db46, State: Initialized, Role: FOLLOWER
I20250905 02:21:20.968799 1693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2679e30a20f242e39dcbfbdd22f8db46" member_type: VOTER }
I20250905 02:21:20.969216 1693 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:20.969400 1693 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:20.969620 1693 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:20.974164 1693 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2679e30a20f242e39dcbfbdd22f8db46" member_type: VOTER }
I20250905 02:21:20.974634 1693 leader_election.cc:304] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2679e30a20f242e39dcbfbdd22f8db46; no voters:
I20250905 02:21:20.975677 1693 leader_election.cc:290] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:20.976114 1696 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:20.977411 1696 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 1 LEADER]: Becoming Leader. State: Replica: 2679e30a20f242e39dcbfbdd22f8db46, State: Running, Role: LEADER
I20250905 02:21:20.978065 1696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2679e30a20f242e39dcbfbdd22f8db46" member_type: VOTER }
I20250905 02:21:20.978576 1693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:20.981226 1698 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2679e30a20f242e39dcbfbdd22f8db46. Latest consensus state: current_term: 1 leader_uuid: "2679e30a20f242e39dcbfbdd22f8db46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2679e30a20f242e39dcbfbdd22f8db46" member_type: VOTER } }
I20250905 02:21:20.982143 1698 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:20.981163 1697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2679e30a20f242e39dcbfbdd22f8db46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2679e30a20f242e39dcbfbdd22f8db46" member_type: VOTER } }
I20250905 02:21:20.982812 1697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:20.985814 1703 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:20.991218 1703 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:20.995062 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:21:21.000032 1703 catalog_manager.cc:1349] Generated new cluster ID: 816d6a2fdb9e4e9dac4949f3844eb8c6
I20250905 02:21:21.000341 1703 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:21.048482 1703 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:21:21.049754 1703 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:21.065824 1703 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46: Generated new TSK 0
I20250905 02:21:21.066431 1703 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:21:21.126717 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:21.134653 1716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.135219 1715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:21.136272 31938 server_base.cc:1047] running on GCE node
W20250905 02:21:21.136432 1718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:21.137463 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:21.137650 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:21.137768 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038881137757 us; error 0 us; skew 500 ppm
I20250905 02:21:21.138198 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:21.140623 31938 webserver.cc:480] Webserver started at http://127.31.48.129:34249/ using document root <none> and password file <none>
I20250905 02:21:21.141072 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:21.141248 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:21.141484 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:21.142724 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "68455ca23ce1475abeacbabeb838a4bc"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-q32q"
I20250905 02:21:21.147225 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 02:21:21.150509 1723 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.151253 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:21:21.151491 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "68455ca23ce1475abeacbabeb838a4bc"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-q32q"
I20250905 02:21:21.151727 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:21.205613 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:21.206848 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:21.208408 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:21.210750 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:21.210938 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.211172 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:21.211333 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.251379 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:40401
I20250905 02:21:21.251461 1785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:40401 every 8 connection(s)
I20250905 02:21:21.256701 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:21.263569 1790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.265424 1791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.268833 1793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:21.269902 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:21.269984 1786 heartbeater.cc:344] Connected to a master server at 127.31.48.190:37101
I20250905 02:21:21.270460 1786 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:21.270766 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:21.270993 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:21.271153 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038881271134 us; error 0 us; skew 500 ppm
I20250905 02:21:21.271235 1786 heartbeater.cc:507] Master 127.31.48.190:37101 requested a full tablet report, sending...
I20250905 02:21:21.271725 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:21.273689 1658 ts_manager.cc:194] Registered new tserver with Master: 68455ca23ce1475abeacbabeb838a4bc (127.31.48.129:40401)
I20250905 02:21:21.274631 31938 webserver.cc:480] Webserver started at http://127.31.48.130:39113/ using document root <none> and password file <none>
I20250905 02:21:21.275277 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:21.275524 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:21.275854 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:21.276611 1658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57516
I20250905 02:21:21.277520 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "57d8b587f4444f9abb3baa82af16413c"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-q32q"
I20250905 02:21:21.282107 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:21:21.285478 1798 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.286326 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:21.286609 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "57d8b587f4444f9abb3baa82af16413c"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-q32q"
I20250905 02:21:21.286867 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:21.306828 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:21.308032 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:21.309455 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:21.311630 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:21.311816 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.312031 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:21.312181 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.351313 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:41379
I20250905 02:21:21.351398 1860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:41379 every 8 connection(s)
I20250905 02:21:21.356223 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:21.363375 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.364636 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:21.367527 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:21.368877 1861 heartbeater.cc:344] Connected to a master server at 127.31.48.190:37101
I20250905 02:21:21.369277 1861 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:21.369266 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:21.370270 1861 heartbeater.cc:507] Master 127.31.48.190:37101 requested a full tablet report, sending...
I20250905 02:21:21.370288 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:21.370657 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:21.370827 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038881370812 us; error 0 us; skew 500 ppm
I20250905 02:21:21.371469 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:21.372166 1658 ts_manager.cc:194] Registered new tserver with Master: 57d8b587f4444f9abb3baa82af16413c (127.31.48.130:41379)
I20250905 02:21:21.373721 1658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57522
I20250905 02:21:21.373868 31938 webserver.cc:480] Webserver started at http://127.31.48.131:44017/ using document root <none> and password file <none>
I20250905 02:21:21.374380 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:21.374640 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:21.374946 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:21.376112 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "47bc6508dc4c408fa9e8fc4ee08d6f16"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-q32q"
I20250905 02:21:21.380259 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:21:21.383899 1872 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.384671 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:21.384938 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "47bc6508dc4c408fa9e8fc4ee08d6f16"
format_stamp: "Formatted at 2025-09-05 02:21:21 on dist-test-slave-q32q"
I20250905 02:21:21.385186 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfNoTablets.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:21.406600 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:21.407824 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:21.409322 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:21.411592 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:21.411808 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.412050 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:21.412209 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:21.451260 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:39359
I20250905 02:21:21.451375 1934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:39359 every 8 connection(s)
I20250905 02:21:21.464978 1935 heartbeater.cc:344] Connected to a master server at 127.31.48.190:37101
I20250905 02:21:21.465612 1935 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:21.466361 1935 heartbeater.cc:507] Master 127.31.48.190:37101 requested a full tablet report, sending...
I20250905 02:21:21.468158 1658 ts_manager.cc:194] Registered new tserver with Master: 47bc6508dc4c408fa9e8fc4ee08d6f16 (127.31.48.131:39359)
I20250905 02:21:21.469234 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014721617s
I20250905 02:21:21.469750 1658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57524
I20250905 02:21:22.279626 1786 heartbeater.cc:499] Master 127.31.48.190:37101 was elected leader, sending a full tablet report...
I20250905 02:21:22.375983 1861 heartbeater.cc:499] Master 127.31.48.190:37101 was elected leader, sending a full tablet report...
I20250905 02:21:22.472653 1935 heartbeater.cc:499] Master 127.31.48.190:37101 was elected leader, sending a full tablet report...
I20250905 02:21:22.994741 1711 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:24.995617 1711 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:26.996708 1711 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:28.997686 1711 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 02:21:29.105978 1782 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3c20 after lost signal to thread 31941
W20250905 02:21:29.106815 1782 debug-util.cc:398] Leaking SignalData structure 0x7b0800010360 after lost signal to thread 1692
W20250905 02:21:29.107666 1782 debug-util.cc:398] Leaking SignalData structure 0x7b08001c7900 after lost signal to thread 1785
W20250905 02:21:29.108662 1782 debug-util.cc:398] Leaking SignalData structure 0x7b08001b1a60 after lost signal to thread 1860
W20250905 02:21:29.109427 1782 debug-util.cc:398] Leaking SignalData structure 0x7b0800190f00 after lost signal to thread 1934
I20250905 02:21:30.998766 1711 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:33.000155 1711 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:33.107259 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:21:33.133495 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:33.166837 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:21:33.175657 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:21:33.199071 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:33.215173 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:21:33.223712 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:21:33.240886 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:33.256914 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:21:33.265511 31938 master.cc:561] Master@127.31.48.190:37101 shutting down...
I20250905 02:21:33.282666 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:33.283454 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:33.283844 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 2679e30a20f242e39dcbfbdd22f8db46: stopping tablet replica
I20250905 02:21:33.302353 31938 master.cc:583] Master@127.31.48.190:37101 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfNoTablets (12474 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne
I20250905 02:21:33.330830 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:41995
I20250905 02:21:33.331974 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:33.337634 1940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.337608 1941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.339282 1943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.339690 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:33.340469 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:33.340664 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:33.340833 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038893340814 us; error 0 us; skew 500 ppm
I20250905 02:21:33.341327 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:33.343662 31938 webserver.cc:480] Webserver started at http://127.31.48.190:45955/ using document root <none> and password file <none>
I20250905 02:21:33.344110 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:33.344290 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:33.344586 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:33.345752 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "3ccc0f23b42a447fb0c9e973f36c79f3"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.350333 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250905 02:21:33.353403 1948 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.354045 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:21:33.354280 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "3ccc0f23b42a447fb0c9e973f36c79f3"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.354543 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:33.370831 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:33.372025 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:33.408214 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:41995
I20250905 02:21:33.408318 1999 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:41995 every 8 connection(s)
I20250905 02:21:33.412166 2000 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:33.423027 2000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3: Bootstrap starting.
I20250905 02:21:33.427389 2000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:33.431352 2000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3: No bootstrap required, opened a new log
I20250905 02:21:33.433354 2000 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" member_type: VOTER }
I20250905 02:21:33.433743 2000 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:33.433943 2000 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccc0f23b42a447fb0c9e973f36c79f3, State: Initialized, Role: FOLLOWER
I20250905 02:21:33.434422 2000 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" member_type: VOTER }
I20250905 02:21:33.434875 2000 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:33.435057 2000 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:33.435341 2000 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:33.439939 2000 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" member_type: VOTER }
I20250905 02:21:33.440557 2000 leader_election.cc:304] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ccc0f23b42a447fb0c9e973f36c79f3; no voters:
I20250905 02:21:33.441679 2000 leader_election.cc:290] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:33.442058 2003 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:33.443403 2003 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccc0f23b42a447fb0c9e973f36c79f3, State: Running, Role: LEADER
I20250905 02:21:33.444020 2003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" member_type: VOTER }
I20250905 02:21:33.444747 2000 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:33.449559 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" member_type: VOTER } }
I20250905 02:21:33.450172 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:33.450551 2005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ccc0f23b42a447fb0c9e973f36c79f3. Latest consensus state: current_term: 1 leader_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccc0f23b42a447fb0c9e973f36c79f3" member_type: VOTER } }
I20250905 02:21:33.451122 2005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:33.452411 2010 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:33.460727 2010 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:33.461730 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:21:33.469869 2010 catalog_manager.cc:1349] Generated new cluster ID: 810b3f34759a4e39ab7da73bc32918c1
I20250905 02:21:33.470158 2010 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:33.492915 2010 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:21:33.494194 2010 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:33.506464 2010 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3: Generated new TSK 0
I20250905 02:21:33.506976 2010 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:21:33.528194 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:33.534320 2021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.535302 2022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.536958 2024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.537680 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:33.538455 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:33.538652 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:33.538791 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038893538779 us; error 0 us; skew 500 ppm
I20250905 02:21:33.539225 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:33.541561 31938 webserver.cc:480] Webserver started at http://127.31.48.129:32813/ using document root <none> and password file <none>
I20250905 02:21:33.541977 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:33.542127 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:33.542341 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:33.543428 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "cb7038251fc940589a3efee833912be5"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.548060 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 02:21:33.551409 2029 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.552235 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:33.552561 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "cb7038251fc940589a3efee833912be5"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.552799 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:33.578052 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:33.579212 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:33.580662 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:33.582885 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:33.583050 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.583304 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:33.583458 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.621867 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:43525
I20250905 02:21:33.621978 2091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:43525 every 8 connection(s)
I20250905 02:21:33.626561 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:33.633198 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.636221 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.637143 31938 server_base.cc:1047] running on GCE node
W20250905 02:21:33.637163 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.638187 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:33.638393 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:33.638551 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038893638532 us; error 0 us; skew 500 ppm
I20250905 02:21:33.638979 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:33.641244 31938 webserver.cc:480] Webserver started at http://127.31.48.130:42443/ using document root <none> and password file <none>
I20250905 02:21:33.641723 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:33.641968 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:33.642285 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:33.643412 2092 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41995
I20250905 02:21:33.643576 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "c683f927812541caa3e10eb84e2223b5"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.643724 2092 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:33.644531 2092 heartbeater.cc:507] Master 127.31.48.190:41995 requested a full tablet report, sending...
I20250905 02:21:33.650130 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20250905 02:21:33.657213 2104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.657380 1965 ts_manager.cc:194] Registered new tserver with Master: cb7038251fc940589a3efee833912be5 (127.31.48.129:43525)
I20250905 02:21:33.658234 31938 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20250905 02:21:33.658540 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "c683f927812541caa3e10eb84e2223b5"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.658816 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:33.659760 1965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52700
I20250905 02:21:33.685854 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:33.687047 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:33.688485 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:33.690673 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:33.690860 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.691087 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:33.691229 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.730681 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:34557
I20250905 02:21:33.730787 2166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:34557 every 8 connection(s)
I20250905 02:21:33.735277 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:33.741743 2170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.743181 2171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.745739 2173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.746302 2167 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41995
I20250905 02:21:33.746637 2167 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:33.746793 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:33.747337 2167 heartbeater.cc:507] Master 127.31.48.190:41995 requested a full tablet report, sending...
I20250905 02:21:33.747666 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:33.747900 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:33.748093 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038893748074 us; error 0 us; skew 500 ppm
I20250905 02:21:33.748673 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:33.749248 1965 ts_manager.cc:194] Registered new tserver with Master: c683f927812541caa3e10eb84e2223b5 (127.31.48.130:34557)
I20250905 02:21:33.750664 1965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52710
I20250905 02:21:33.751084 31938 webserver.cc:480] Webserver started at http://127.31.48.131:44059/ using document root <none> and password file <none>
I20250905 02:21:33.751559 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:33.751758 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:33.752002 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:33.753180 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "98cfd33fb2674a78a748d2be83c312d6"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.757892 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:21:33.761179 2178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.761936 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:33.762230 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "98cfd33fb2674a78a748d2be83c312d6"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.762511 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:33.776087 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:33.777206 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:33.778605 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:33.780786 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:33.780977 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.781204 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:33.781353 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.818920 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:40269
I20250905 02:21:33.819020 2240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:40269 every 8 connection(s)
I20250905 02:21:33.823637 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:33.830565 2244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:33.832182 2245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.835601 31938 server_base.cc:1047] running on GCE node
W20250905 02:21:33.835521 2247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:33.836491 2241 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41995
I20250905 02:21:33.836818 2241 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:33.837466 2241 heartbeater.cc:507] Master 127.31.48.190:41995 requested a full tablet report, sending...
I20250905 02:21:33.838743 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:33.838990 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:33.839181 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038893839150 us; error 0 us; skew 500 ppm
I20250905 02:21:33.839403 1965 ts_manager.cc:194] Registered new tserver with Master: 98cfd33fb2674a78a748d2be83c312d6 (127.31.48.131:40269)
I20250905 02:21:33.839699 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:33.840963 1965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52726
I20250905 02:21:33.842203 31938 webserver.cc:480] Webserver started at http://127.31.48.132:40749/ using document root <none> and password file <none>
I20250905 02:21:33.842667 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:33.842854 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:33.843086 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:33.844158 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "2f48b238e70d4517bbb066b9c68745fc"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.848208 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20250905 02:21:33.851187 2252 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.851908 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:21:33.852180 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "2f48b238e70d4517bbb066b9c68745fc"
format_stamp: "Formatted at 2025-09-05 02:21:33 on dist-test-slave-q32q"
I20250905 02:21:33.852500 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:33.872859 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:33.874123 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:33.875653 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:33.878115 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:33.878310 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.878536 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:33.878686 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:33.918128 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:43471
I20250905 02:21:33.918226 2314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:43471 every 8 connection(s)
I20250905 02:21:33.931635 2315 heartbeater.cc:344] Connected to a master server at 127.31.48.190:41995
I20250905 02:21:33.931973 2315 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:33.932690 2315 heartbeater.cc:507] Master 127.31.48.190:41995 requested a full tablet report, sending...
I20250905 02:21:33.934463 1965 ts_manager.cc:194] Registered new tserver with Master: 2f48b238e70d4517bbb066b9c68745fc (127.31.48.132:43471)
I20250905 02:21:33.934952 31938 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.013630413s
I20250905 02:21:33.935812 1965 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52732
==================
WARNING: ThreadSanitizer: data race (pid=31938)
Write of size 1 at 0x7b48002c0fc0 by main thread:
#0 void std::__1::__optional_storage_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__construct<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:324:26 (auto_rebalancer-test+0x3959af)
#1 std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >& std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::operator=<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:790:19 (auto_rebalancer-test+0x39588e)
#2 kudu::master::TSDescriptor::AssignLocationForTesting(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/ts_descriptor.h:278:15 (auto_rebalancer-test+0x39375f)
#3 kudu::master::AutoRebalancerTest::AssignLocationsEvenly(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:140:17 (auto_rebalancer-test+0x3816f1)
#4 kudu::master::AutoRebalancerTest_NoReplicaMovesIfLocationLoadSkewedByOne_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:460:3 (auto_rebalancer-test+0x36ca46)
#5 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#6 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#7 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#8 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#9 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#10 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#11 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#12 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#13 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#14 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (auto_rebalancer-test+0x3ac43b)
#15 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:115:10 (auto_rebalancer-test+0x3ab07c)
Previous read of size 1 at 0x7b48002c0fc0 by thread T389 (mutexes: read M521990155992305312):
#0 std::__1::__optional_storage_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::has_value() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:294:22 (libksck.so+0x10fb7a)
#1 void std::__1::__optional_storage_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__construct_from<std::__1::__optional_copy_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false> const&>(std::__1::__optional_copy_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:331:19 (libksck.so+0x10ff6d)
#2 std::__1::__optional_copy_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__optional_copy_base(std::__1::__optional_copy_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:464:15 (libmaster.so+0x2e3948)
#3 std::__1::__optional_move_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__optional_move_base(std::__1::__optional_move_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:490:5 (libmaster.so+0x2e3900)
#4 std::__1::__optional_copy_assign_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__optional_copy_assign_base(std::__1::__optional_copy_assign_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:522:5 (libmaster.so+0x2e38c0)
#5 std::__1::__optional_move_assign_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__optional_move_assign_base(std::__1::__optional_move_assign_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false> const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:555:5 (libmaster.so+0x2e3880)
#6 std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >::optional(std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:688:41 (libmaster.so+0x2e3570)
#7 kudu::master::TSDescriptor::location() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/ts_descriptor.h:250:12 (libmaster.so+0x2d6c28)
#8 kudu::master::AutoRebalancerTask::BuildClusterRawInfo(std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, kudu::rebalance::ClusterRawInfo*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer.cc:494:13 (libmaster.so+0x2cf9d0)
#9 kudu::master::AutoRebalancerTask::RunLoop() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer.cc:225:16 (libmaster.so+0x2cf002)
#10 kudu::master::AutoRebalancerTask::Init()::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer.cc:185:42 (libmaster.so+0x2d41d1)
#11 decltype(std::__1::forward<kudu::master::AutoRebalancerTask::Init()::$_0&>(fp)()) std::__1::__invoke<kudu::master::AutoRebalancerTask::Init()::$_0&>(kudu::master::AutoRebalancerTask::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3905:1 (libmaster.so+0x2d4189)
#12 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::master::AutoRebalancerTask::Init()::$_0&>(kudu::master::AutoRebalancerTask::Init()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libmaster.so+0x2d4119)
#13 std::__1::__function::__alloc_func<kudu::master::AutoRebalancerTask::Init()::$_0, std::__1::allocator<kudu::master::AutoRebalancerTask::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libmaster.so+0x2d40e1)
#14 std::__1::__function::__func<kudu::master::AutoRebalancerTask::Init()::$_0, std::__1::allocator<kudu::master::AutoRebalancerTask::Init()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libmaster.so+0x2d33dd)
#15 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (auto_rebalancer-test+0x3ac6d4)
#16 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (auto_rebalancer-test+0x3ac519)
#17 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x44676a)
Location is heap block of size 376 at 0x7b48002c0e80 allocated by thread T425:
#0 operator new(unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:64 (auto_rebalancer-test+0x3695a7)
#1 std::__1::__libcpp_allocate(unsigned long, unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/new:253:10 (libmaster.so+0x2c4d86)
#2 std::__1::allocator<std::__1::__shared_ptr_emplace<std::__1::shared_ptr<kudu::master::TSDescriptor> enable_make_shared<kudu::master::TSDescriptor>::make_shared<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::make_shared_enabler, std::__1::allocator<std::__1::shared_ptr<kudu::master::TSDescriptor> enable_make_shared<kudu::master::TSDescriptor>::make_shared<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::make_shared_enabler> > >::allocate(unsigned long) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:1789:34 (libmaster.so+0x4a6da1)
#3 std::__1::enable_if<!(is_array<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>::value), std::__1::shared_ptr<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&> >::type std::__1::make_shared<std::__1::shared_ptr<kudu::master::TSDescriptor> enable_make_shared<kudu::master::TSDescriptor>::make_shared<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::make_shared_enabler, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::shared_ptr<kudu::master::TSDescriptor> enable_make_shared<kudu::master::TSDescriptor>::make_shared<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::make_shared_enabler&&...) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/memory:4290:45 (libmaster.so+0x4a6bd9)
#4 std::__1::shared_ptr<kudu::master::TSDescriptor> enable_make_shared<kudu::master::TSDescriptor>::make_shared<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/make_shared.h:61:12 (libmaster.so+0x4a2ced)
#5 kudu::master::TSDescriptor::RegisterNew(kudu::NodeInstancePB const&, kudu::ServerRegistrationPB const&, std::__1::optional<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > const&, kudu::DnsResolver*, std::__1::shared_ptr<kudu::master::TSDescriptor>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/ts_descriptor.cc:72:32 (libmaster.so+0x4a02c0)
#6 kudu::master::TSManager::RegisterTS(kudu::NodeInstancePB const&, kudu::ServerRegistrationPB const&, kudu::DnsResolver*, std::__1::shared_ptr<kudu::master::TSDescriptor>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/ts_manager.cc:188:7 (libmaster.so+0x4b504b)
#7 kudu::master::MasterServiceImpl::TSHeartbeat(kudu::master::TSHeartbeatRequestPB const*, kudu::master::TSHeartbeatResponsePB*, kudu::rpc::RpcContext*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master_service.cc:418:39 (libmaster.so+0x437ca4)
#8 kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/build/tsan/src/kudu/master/master.service.cc:585:13 (libmaster_proto.so+0x249484)
#9 decltype(std::__1::forward<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1&>(fp)(std::__1::forward<google::protobuf::Message const*>(fp0), std::__1::forward<google::protobuf::Message*>(fp0), std::__1::forward<kudu::rpc::RpcContext*>(fp0))) std::__1::__invoke<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3905:1 (libmaster_proto.so+0x249412)
#10 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*>(kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libmaster_proto.so+0x249341)
#11 std::__1::__function::__alloc_func<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1, std::__1::allocator<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libmaster_proto.so+0x2492bc)
#12 std::__1::__function::__func<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1, std::__1::allocator<kudu::master::MasterServiceIf::MasterServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_1>, void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libmaster_proto.so+0x248572)
#13 std::__1::__function::__value_func<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (libkrpc.so+0x1f3ddc)
#14 std::__1::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (libkrpc.so+0x1f3216)
#15 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_if.cc:137:3 (libkrpc.so+0x1f2bbf)
#16 kudu::rpc::ServicePool::RunThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:229:15 (libkrpc.so+0x1f5ed3)
#17 kudu::rpc::ServicePool::Init(int)::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0x1f71f1)
#18 decltype(std::__1::forward<kudu::rpc::ServicePool::Init(int)::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ServicePool::Init(int)::$_0&>(kudu::rpc::ServicePool::Init(int)::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3905:1 (libkrpc.so+0x1f71a9)
#19 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::rpc::ServicePool::Init(int)::$_0&>(kudu::rpc::ServicePool::Init(int)::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkrpc.so+0x1f7139)
#20 std::__1::__function::__alloc_func<kudu::rpc::ServicePool::Init(int)::$_0, std::__1::allocator<kudu::rpc::ServicePool::Init(int)::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkrpc.so+0x1f7101)
#21 std::__1::__function::__func<kudu::rpc::ServicePool::Init(int)::$_0, std::__1::allocator<kudu::rpc::ServicePool::Init(int)::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkrpc.so+0x1f63fd)
#22 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (auto_rebalancer-test+0x3ac6d4)
#23 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (auto_rebalancer-test+0x3ac519)
#24 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x44676a)
Mutex M521990155992305312 is already destroyed.
Thread T389 'auto-rebalancer' (tid=2013, running) created by thread T76 at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (auto_rebalancer-test+0x2edbb5)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15 (libkudu_util.so+0x446018)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12 (libmaster.so+0x2d4b99)
#3 kudu::master::AutoRebalancerTask::Init() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer.cc:184:10 (libmaster.so+0x2ced12)
#4 kudu::master::CatalogManager::Init(bool) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:1020:3 (libmaster.so+0x310cec)
#5 kudu::master::Master::InitCatalogManager() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master.cc:426:3 (libmaster.so+0x3f97b5)
#6 kudu::master::Master::InitCatalogManagerTask() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master.cc:414:14 (libmaster.so+0x3f9613)
#7 kudu::master::Master::StartAsync()::$_0::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master.cc:394:3 (libmaster.so+0x3fdfb1)
#8 decltype(std::__1::forward<kudu::master::Master::StartAsync()::$_0&>(fp)()) std::__1::__invoke<kudu::master::Master::StartAsync()::$_0&>(kudu::master::Master::StartAsync()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3905:1 (libmaster.so+0x3fdf69)
#9 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::master::Master::StartAsync()::$_0&>(kudu::master::Master::StartAsync()::$_0&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libmaster.so+0x3fdef9)
#10 std::__1::__function::__alloc_func<kudu::master::Master::StartAsync()::$_0, std::__1::allocator<kudu::master::Master::StartAsync()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libmaster.so+0x3fdec1)
#11 std::__1::__function::__func<kudu::master::Master::StartAsync()::$_0, std::__1::allocator<kudu::master::Master::StartAsync()::$_0>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libmaster.so+0x3fd1bd)
#12 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (auto_rebalancer-test+0x3ac6d4)
#13 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (auto_rebalancer-test+0x3ac519)
#14 kudu::ThreadPool::DispatchThread() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:733:7 (libkudu_util.so+0x463464)
#15 kudu::ThreadPool::CreateThread()::$_1::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/threadpool.cc:815:48 (libkudu_util.so+0x4657b1)
#16 decltype(std::__1::forward<kudu::ThreadPool::CreateThread()::$_1&>(fp)()) std::__1::__invoke<kudu::ThreadPool::CreateThread()::$_1&>(kudu::ThreadPool::CreateThread()::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3905:1 (libkudu_util.so+0x465769)
#17 void std::__1::__invoke_void_return_wrapper<void>::__call<kudu::ThreadPool::CreateThread()::$_1&>(kudu::ThreadPool::CreateThread()::$_1&) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/__functional_base:348:9 (libkudu_util.so+0x4656f9)
#18 std::__1::__function::__alloc_func<kudu::ThreadPool::CreateThread()::$_1, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1557:16 (libkudu_util.so+0x4656c1)
#19 std::__1::__function::__func<kudu::ThreadPool::CreateThread()::$_1, std::__1::allocator<kudu::ThreadPool::CreateThread()::$_1>, void ()>::operator()() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1731:12 (libkudu_util.so+0x4649bd)
#20 std::__1::__function::__value_func<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:1884:16 (auto_rebalancer-test+0x3ac6d4)
#21 std::__1::function<void ()>::operator()() const /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/functional:2556:12 (auto_rebalancer-test+0x3ac519)
#22 kudu::Thread::SuperviseThread(void*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:694:3 (libkudu_util.so+0x44676a)
Thread T425 'rpc worker-1965' (tid=1965, running) created by main thread at:
#0 pthread_create /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:966 (auto_rebalancer-test+0x2edbb5)
#1 kudu::Thread::StartThread(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, unsigned long, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:643:15 (libkudu_util.so+0x446018)
#2 kudu::Thread::Create(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::function<void ()>, scoped_refptr<kudu::Thread>*) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.h:147:12 (libmaster.so+0x2d4b99)
#3 kudu::rpc::ServicePool::Init(int) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/service_pool.cc:92:5 (libkrpc.so+0x1f4eff)
#4 kudu::RpcServer::RegisterService(std::__1::unique_ptr<kudu::rpc::ServiceIf, std::__1::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/rpc_server.cc:238:3 (libserver_process.so+0x1354cf)
#5 kudu::server::ServerBase::RegisterService(std::__1::unique_ptr<kudu::rpc::ServiceIf, std::__1::default_delete<kudu::rpc::ServiceIf> >) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/server/server_base.cc:1184:23 (libserver_process.so+0x1479bc)
#6 kudu::master::Master::StartAsync() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/master.cc:382:3 (libmaster.so+0x3f8727)
#7 kudu::master::MiniMaster::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/mini_master.cc:100:3 (libmaster.so+0x4c9e22)
#8 kudu::cluster::InternalMiniCluster::StartMasters() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/internal_mini_cluster.cc:178:5 (libmini_cluster.so+0xd6edf)
#9 kudu::cluster::InternalMiniCluster::Start() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/mini-cluster/internal_mini_cluster.cc:109:3 (libmini_cluster.so+0xd670b)
#10 kudu::master::AutoRebalancerTest::CreateAndStartCluster(bool) /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:118:22 (auto_rebalancer-test+0x3806c8)
#11 kudu::master::AutoRebalancerTest_NoReplicaMovesIfLocationLoadSkewedByOne_Test::TestBody() /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:457:3 (auto_rebalancer-test+0x36c869)
#12 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x64e2f)
#13 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x64e2f)
#14 testing::Test::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2674:5 (libgtest.so.1.12.1+0x42a31)
#15 testing::TestInfo::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2853:11 (libgtest.so.1.12.1+0x43d48)
#16 testing::TestSuite::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:3012:30 (libgtest.so.1.12.1+0x44d24)
#17 testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x59814)
#18 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x65cef)
#19 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x65cef)
#20 testing::UnitTest::Run() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/googletest-release-1.12.1/googletest/src/gtest.cc:5444:10 (libgtest.so.1.12.1+0x58dcc)
#21 RUN_ALL_TESTS() /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/gtest/gtest.h:2293:73 (auto_rebalancer-test+0x3ac43b)
#22 main /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_main.cc:115:10 (auto_rebalancer-test+0x3ab07c)
SUMMARY: ThreadSanitizer: data race /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/optional:324:26 in void std::__1::__optional_storage_base<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, false>::__construct<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&&)
==================
I20250905 02:21:34.556257 31938 test_util.cc:276] Using random seed: 2133501002
I20250905 02:21:34.577988 1965 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52744:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 02:21:34.627282 2057 tablet_service.cc:1468] Processing CreateTablet for tablet cbf55d2cce8a4433bccb9ef3b5510ede (DEFAULT_TABLE table=test-workload [id=c1f75054e85849c098bb56cec2c0cf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:34.628062 2132 tablet_service.cc:1468] Processing CreateTablet for tablet cbf55d2cce8a4433bccb9ef3b5510ede (DEFAULT_TABLE table=test-workload [id=c1f75054e85849c098bb56cec2c0cf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:34.628786 2057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf55d2cce8a4433bccb9ef3b5510ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:34.629313 2132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf55d2cce8a4433bccb9ef3b5510ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:34.630731 2280 tablet_service.cc:1468] Processing CreateTablet for tablet cbf55d2cce8a4433bccb9ef3b5510ede (DEFAULT_TABLE table=test-workload [id=c1f75054e85849c098bb56cec2c0cf2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:34.632017 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf55d2cce8a4433bccb9ef3b5510ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:34.646519 2336 tablet_bootstrap.cc:492] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5: Bootstrap starting.
I20250905 02:21:34.649592 2337 tablet_bootstrap.cc:492] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: Bootstrap starting.
I20250905 02:21:34.655229 2336 tablet_bootstrap.cc:654] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:34.656941 2337 tablet_bootstrap.cc:654] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:34.661901 2338 tablet_bootstrap.cc:492] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc: Bootstrap starting.
I20250905 02:21:34.662973 2092 heartbeater.cc:499] Master 127.31.48.190:41995 was elected leader, sending a full tablet report...
I20250905 02:21:34.670544 2338 tablet_bootstrap.cc:654] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:34.670729 2337 tablet_bootstrap.cc:492] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: No bootstrap required, opened a new log
I20250905 02:21:34.671139 2336 tablet_bootstrap.cc:492] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5: No bootstrap required, opened a new log
I20250905 02:21:34.671249 2337 ts_tablet_manager.cc:1397] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.006s
I20250905 02:21:34.671630 2336 ts_tablet_manager.cc:1397] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20250905 02:21:34.674275 2337 raft_consensus.cc:357] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.674454 2336 raft_consensus.cc:357] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.675122 2337 raft_consensus.cc:383] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:34.675494 2338 tablet_bootstrap.cc:492] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc: No bootstrap required, opened a new log
I20250905 02:21:34.675603 2337 raft_consensus.cc:738] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c683f927812541caa3e10eb84e2223b5, State: Initialized, Role: FOLLOWER
I20250905 02:21:34.675956 2338 ts_tablet_manager.cc:1397] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20250905 02:21:34.675172 2336 raft_consensus.cc:383] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:34.676431 2336 raft_consensus.cc:738] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb7038251fc940589a3efee833912be5, State: Initialized, Role: FOLLOWER
I20250905 02:21:34.677142 2336 consensus_queue.cc:260] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.677137 2337 consensus_queue.cc:260] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.678447 2338 raft_consensus.cc:357] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.679145 2338 raft_consensus.cc:383] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:34.679508 2338 raft_consensus.cc:738] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f48b238e70d4517bbb066b9c68745fc, State: Initialized, Role: FOLLOWER
I20250905 02:21:34.680392 2338 consensus_queue.cc:260] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.680604 2336 ts_tablet_manager.cc:1428] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20250905 02:21:34.681046 2337 ts_tablet_manager.cc:1428] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20250905 02:21:34.680626 2167 heartbeater.cc:499] Master 127.31.48.190:41995 was elected leader, sending a full tablet report...
I20250905 02:21:34.682766 2315 heartbeater.cc:499] Master 127.31.48.190:41995 was elected leader, sending a full tablet report...
I20250905 02:21:34.683439 2338 ts_tablet_manager.cc:1428] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
W20250905 02:21:34.735298 2168 tablet.cc:2378] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:21:34.843441 2241 heartbeater.cc:499] Master 127.31.48.190:41995 was elected leader, sending a full tablet report...
I20250905 02:21:34.847638 2344 raft_consensus.cc:491] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:21:34.848076 2344 raft_consensus.cc:513] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.850008 2344 leader_election.cc:290] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cb7038251fc940589a3efee833912be5 (127.31.48.129:43525), c683f927812541caa3e10eb84e2223b5 (127.31.48.130:34557)
I20250905 02:21:34.861997 2142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf55d2cce8a4433bccb9ef3b5510ede" candidate_uuid: "2f48b238e70d4517bbb066b9c68745fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c683f927812541caa3e10eb84e2223b5" is_pre_election: true
I20250905 02:21:34.862615 2142 raft_consensus.cc:2466] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f48b238e70d4517bbb066b9c68745fc in term 0.
I20250905 02:21:34.863595 2067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf55d2cce8a4433bccb9ef3b5510ede" candidate_uuid: "2f48b238e70d4517bbb066b9c68745fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7038251fc940589a3efee833912be5" is_pre_election: true
I20250905 02:21:34.863651 2255 leader_election.cc:304] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f48b238e70d4517bbb066b9c68745fc, c683f927812541caa3e10eb84e2223b5; no voters:
I20250905 02:21:34.864189 2067 raft_consensus.cc:2466] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f48b238e70d4517bbb066b9c68745fc in term 0.
I20250905 02:21:34.864467 2344 raft_consensus.cc:2802] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:21:34.864820 2344 raft_consensus.cc:491] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:21:34.865159 2344 raft_consensus.cc:3058] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:34.869921 2344 raft_consensus.cc:513] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.871295 2344 leader_election.cc:290] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [CANDIDATE]: Term 1 election: Requested vote from peers cb7038251fc940589a3efee833912be5 (127.31.48.129:43525), c683f927812541caa3e10eb84e2223b5 (127.31.48.130:34557)
I20250905 02:21:34.872282 2142 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf55d2cce8a4433bccb9ef3b5510ede" candidate_uuid: "2f48b238e70d4517bbb066b9c68745fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c683f927812541caa3e10eb84e2223b5"
I20250905 02:21:34.872277 2067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf55d2cce8a4433bccb9ef3b5510ede" candidate_uuid: "2f48b238e70d4517bbb066b9c68745fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb7038251fc940589a3efee833912be5"
I20250905 02:21:34.872925 2142 raft_consensus.cc:3058] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:34.872972 2067 raft_consensus.cc:3058] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:34.877720 2142 raft_consensus.cc:2466] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f48b238e70d4517bbb066b9c68745fc in term 1.
I20250905 02:21:34.877720 2067 raft_consensus.cc:2466] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f48b238e70d4517bbb066b9c68745fc in term 1.
I20250905 02:21:34.878844 2255 leader_election.cc:304] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f48b238e70d4517bbb066b9c68745fc, c683f927812541caa3e10eb84e2223b5; no voters:
I20250905 02:21:34.879484 2344 raft_consensus.cc:2802] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:34.880548 2344 raft_consensus.cc:695] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 1 LEADER]: Becoming Leader. State: Replica: 2f48b238e70d4517bbb066b9c68745fc, State: Running, Role: LEADER
I20250905 02:21:34.881212 2344 consensus_queue.cc:237] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } }
I20250905 02:21:34.888248 1964 catalog_manager.cc:5582] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f48b238e70d4517bbb066b9c68745fc (127.31.48.132). New cstate: current_term: 1 leader_uuid: "2f48b238e70d4517bbb066b9c68745fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f48b238e70d4517bbb066b9c68745fc" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 43471 } health_report { overall_health: HEALTHY } } }
I20250905 02:21:35.320310 2344 consensus_queue.cc:1035] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c683f927812541caa3e10eb84e2223b5" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 34557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:35.334460 2344 consensus_queue.cc:1035] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb7038251fc940589a3efee833912be5" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 43525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:35.460638 2018 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:35.461938 2018 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:35.462249 2018 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:37.463196 2018 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:37.464504 2018 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:37.464792 2018 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:39.465670 2018 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:39.466939 2018 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:39.467247 2018 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:41.468334 2018 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:41.470055 2018 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:41.470345 2018 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:43.471189 2018 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:43.472419 2018 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:43.472695 2018 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:45.473652 2018 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:45.474903 2018 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:45.475206 2018 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:45.994071 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:21:46.017508 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:46.018201 31938 tablet_replica.cc:331] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5: stopping tablet replica
I20250905 02:21:46.018784 31938 raft_consensus.cc:2241] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:46.019340 31938 raft_consensus.cc:2270] T cbf55d2cce8a4433bccb9ef3b5510ede P cb7038251fc940589a3efee833912be5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:46.040144 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:21:46.051223 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
W20250905 02:21:46.074760 2254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:43525: connect: Connection refused (error 111)
W20250905 02:21:46.080632 2254 consensus_peers.cc:489] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc -> Peer cb7038251fc940589a3efee833912be5 (127.31.48.129:43525): Couldn't send request to peer cb7038251fc940589a3efee833912be5. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:43525: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:46.081974 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:46.082567 31938 tablet_replica.cc:331] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5: stopping tablet replica
I20250905 02:21:46.083070 31938 raft_consensus.cc:2241] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:46.083505 31938 raft_consensus.cc:2270] T cbf55d2cce8a4433bccb9ef3b5510ede P c683f927812541caa3e10eb84e2223b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:46.102135 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:21:46.111960 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:21:46.132297 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
W20250905 02:21:46.134723 2255 consensus_peers.cc:489] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc -> Peer c683f927812541caa3e10eb84e2223b5 (127.31.48.130:34557): Couldn't send request to peer c683f927812541caa3e10eb84e2223b5. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.130:34557: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:21:46.148397 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:21:46.156901 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:21:46.175633 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:46.176218 31938 tablet_replica.cc:331] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc: stopping tablet replica
I20250905 02:21:46.176728 31938 raft_consensus.cc:2241] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:46.177572 31938 raft_consensus.cc:2270] T cbf55d2cce8a4433bccb9ef3b5510ede P 2f48b238e70d4517bbb066b9c68745fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:46.196321 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:21:46.206249 31938 master.cc:561] Master@127.31.48.190:41995 shutting down...
I20250905 02:21:46.223450 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:46.224056 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:46.224344 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 3ccc0f23b42a447fb0c9e973f36c79f3: stopping tablet replica
I20250905 02:21:46.242417 31938 master.cc:583] Master@127.31.48.190:41995 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfLocationLoadSkewedByOne (12943 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy
I20250905 02:21:46.275156 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:39685
I20250905 02:21:46.276280 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:46.281929 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.282126 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.283223 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:46.284399 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:46.285102 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:46.285292 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:46.285552 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038906285529 us; error 0 us; skew 500 ppm
I20250905 02:21:46.286193 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:46.288565 31938 webserver.cc:480] Webserver started at http://127.31.48.190:35297/ using document root <none> and password file <none>
I20250905 02:21:46.289031 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:46.289233 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:46.289479 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:46.290608 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "abdec3ca9713415d9367a94e4f87e76b"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.295148 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:21:46.298391 2371 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.299093 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:21:46.299368 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "abdec3ca9713415d9367a94e4f87e76b"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.299618 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:46.315295 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:46.316583 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:46.353822 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:39685
I20250905 02:21:46.353924 2422 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:39685 every 8 connection(s)
I20250905 02:21:46.357653 2423 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:46.369442 2423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b: Bootstrap starting.
I20250905 02:21:46.374130 2423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:46.378326 2423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b: No bootstrap required, opened a new log
I20250905 02:21:46.380345 2423 raft_consensus.cc:357] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdec3ca9713415d9367a94e4f87e76b" member_type: VOTER }
I20250905 02:21:46.380834 2423 raft_consensus.cc:383] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:46.381021 2423 raft_consensus.cc:738] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdec3ca9713415d9367a94e4f87e76b, State: Initialized, Role: FOLLOWER
I20250905 02:21:46.381553 2423 consensus_queue.cc:260] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdec3ca9713415d9367a94e4f87e76b" member_type: VOTER }
I20250905 02:21:46.382019 2423 raft_consensus.cc:397] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:21:46.382203 2423 raft_consensus.cc:491] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:21:46.382438 2423 raft_consensus.cc:3058] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:46.386829 2423 raft_consensus.cc:513] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdec3ca9713415d9367a94e4f87e76b" member_type: VOTER }
I20250905 02:21:46.387301 2423 leader_election.cc:304] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abdec3ca9713415d9367a94e4f87e76b; no voters:
I20250905 02:21:46.388475 2423 leader_election.cc:290] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:21:46.388851 2426 raft_consensus.cc:2802] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:46.390241 2426 raft_consensus.cc:695] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 1 LEADER]: Becoming Leader. State: Replica: abdec3ca9713415d9367a94e4f87e76b, State: Running, Role: LEADER
I20250905 02:21:46.390954 2426 consensus_queue.cc:237] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdec3ca9713415d9367a94e4f87e76b" member_type: VOTER }
I20250905 02:21:46.391412 2423 sys_catalog.cc:564] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:21:46.393808 2427 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abdec3ca9713415d9367a94e4f87e76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdec3ca9713415d9367a94e4f87e76b" member_type: VOTER } }
I20250905 02:21:46.393819 2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [sys.catalog]: SysCatalogTable state changed. Reason: New leader abdec3ca9713415d9367a94e4f87e76b. Latest consensus state: current_term: 1 leader_uuid: "abdec3ca9713415d9367a94e4f87e76b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abdec3ca9713415d9367a94e4f87e76b" member_type: VOTER } }
I20250905 02:21:46.394495 2427 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:46.394678 2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [sys.catalog]: This master's current role is: LEADER
I20250905 02:21:46.403367 2431 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:21:46.410929 2431 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:21:46.411973 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:21:46.419790 2431 catalog_manager.cc:1349] Generated new cluster ID: fa763be6ae0a49db945bf252158cceb7
I20250905 02:21:46.420096 2431 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:21:46.439280 2431 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:21:46.440590 2431 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:21:46.452667 2431 catalog_manager.cc:5955] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b: Generated new TSK 0
I20250905 02:21:46.453245 2431 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:21:46.479163 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:46.484920 2444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.486433 2445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:46.487948 31938 server_base.cc:1047] running on GCE node
W20250905 02:21:46.487854 2447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:46.489020 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:46.489210 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:46.489342 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038906489321 us; error 0 us; skew 500 ppm
I20250905 02:21:46.489740 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:46.491834 31938 webserver.cc:480] Webserver started at http://127.31.48.129:35829/ using document root <none> and password file <none>
I20250905 02:21:46.492225 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:46.492419 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:46.492645 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:46.493652 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "26e2564f449540c8a2548e3cc047729c"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.497596 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20250905 02:21:46.500470 2452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.501179 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250905 02:21:46.501456 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "26e2564f449540c8a2548e3cc047729c"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.501703 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:46.524752 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:46.525861 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:46.527253 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:46.529456 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:46.529656 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.529892 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:46.530064 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.567826 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:46807
I20250905 02:21:46.567919 2514 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:46807 every 8 connection(s)
I20250905 02:21:46.572680 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:46.578838 2519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.579380 2520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.582113 2523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:46.583173 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:46.584079 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 02:21:46.584147 2515 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39685
W20250905 02:21:46.584267 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:46.584615 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038906584596 us; error 0 us; skew 500 ppm
I20250905 02:21:46.584642 2515 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:46.585165 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:46.585386 2515 heartbeater.cc:507] Master 127.31.48.190:39685 requested a full tablet report, sending...
I20250905 02:21:46.587422 2388 ts_manager.cc:194] Registered new tserver with Master: 26e2564f449540c8a2548e3cc047729c (127.31.48.129:46807)
I20250905 02:21:46.587555 31938 webserver.cc:480] Webserver started at http://127.31.48.130:42387/ using document root <none> and password file <none>
I20250905 02:21:46.588078 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:46.588267 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:46.588524 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:46.589231 2388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55206
I20250905 02:21:46.589726 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "04a403bb917c4ffa8f4be308c1dea187"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.594082 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:21:46.597209 2527 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.598007 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:46.598237 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "04a403bb917c4ffa8f4be308c1dea187"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.598474 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:46.621234 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:46.622349 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:46.623795 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:46.625970 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:46.626168 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.626420 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:46.626562 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.664144 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:46163
I20250905 02:21:46.664245 2589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:46163 every 8 connection(s)
I20250905 02:21:46.668817 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:21:46.675732 2593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.676916 2594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:21:46.679472 2596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:21:46.679955 31938 server_base.cc:1047] running on GCE node
I20250905 02:21:46.680959 2590 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39685
I20250905 02:21:46.680931 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:21:46.681417 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:21:46.681493 2590 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:46.681681 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038906681661 us; error 0 us; skew 500 ppm
I20250905 02:21:46.682394 2590 heartbeater.cc:507] Master 127.31.48.190:39685 requested a full tablet report, sending...
I20250905 02:21:46.682483 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:21:46.684549 2388 ts_manager.cc:194] Registered new tserver with Master: 04a403bb917c4ffa8f4be308c1dea187 (127.31.48.130:46163)
I20250905 02:21:46.685114 31938 webserver.cc:480] Webserver started at http://127.31.48.131:41967/ using document root <none> and password file <none>
I20250905 02:21:46.685588 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:21:46.685770 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:21:46.685935 2388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55210
I20250905 02:21:46.686028 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:21:46.687613 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "fbe9e2ddd60246a7b7b650181e1df954"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.692101 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:21:46.695457 2601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.696166 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:21:46.696465 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "fbe9e2ddd60246a7b7b650181e1df954"
format_stamp: "Formatted at 2025-09-05 02:21:46 on dist-test-slave-q32q"
I20250905 02:21:46.696708 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:21:46.711158 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:21:46.712252 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:21:46.713675 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:21:46.715706 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:21:46.715883 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.716054 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:21:46.716183 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:21:46.753587 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:33731
I20250905 02:21:46.753674 2663 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:33731 every 8 connection(s)
I20250905 02:21:46.766000 2664 heartbeater.cc:344] Connected to a master server at 127.31.48.190:39685
I20250905 02:21:46.766360 2664 heartbeater.cc:461] Registering TS with master...
I20250905 02:21:46.767051 2664 heartbeater.cc:507] Master 127.31.48.190:39685 requested a full tablet report, sending...
I20250905 02:21:46.768848 2388 ts_manager.cc:194] Registered new tserver with Master: fbe9e2ddd60246a7b7b650181e1df954 (127.31.48.131:33731)
I20250905 02:21:46.768963 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01241336s
I20250905 02:21:46.770136 2388 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55212
I20250905 02:21:47.591549 2515 heartbeater.cc:499] Master 127.31.48.190:39685 was elected leader, sending a full tablet report...
I20250905 02:21:47.688653 2590 heartbeater.cc:499] Master 127.31.48.190:39685 was elected leader, sending a full tablet report...
I20250905 02:21:47.772526 2664 heartbeater.cc:499] Master 127.31.48.190:39685 was elected leader, sending a full tablet report...
I20250905 02:21:47.817009 31938 test_util.cc:276] Using random seed: 2146761755
I20250905 02:21:47.836845 2388 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55224:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:21:47.838681 2388 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:21:47.885764 2629 tablet_service.cc:1468] Processing CreateTablet for tablet e5b0e019946d418ab8decde40e2e3771 (DEFAULT_TABLE table=test-workload [id=847e7f70d69345efb2d4cbc6130f953e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:47.886260 2480 tablet_service.cc:1468] Processing CreateTablet for tablet e5b0e019946d418ab8decde40e2e3771 (DEFAULT_TABLE table=test-workload [id=847e7f70d69345efb2d4cbc6130f953e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:47.887167 2629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5b0e019946d418ab8decde40e2e3771. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:47.887403 2480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5b0e019946d418ab8decde40e2e3771. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:47.901165 2555 tablet_service.cc:1468] Processing CreateTablet for tablet e5b0e019946d418ab8decde40e2e3771 (DEFAULT_TABLE table=test-workload [id=847e7f70d69345efb2d4cbc6130f953e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:21:47.902477 2555 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5b0e019946d418ab8decde40e2e3771. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:21:47.905089 2686 tablet_bootstrap.cc:492] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954: Bootstrap starting.
I20250905 02:21:47.915166 2686 tablet_bootstrap.cc:654] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:47.922808 2686 tablet_bootstrap.cc:492] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954: No bootstrap required, opened a new log
I20250905 02:21:47.923406 2686 ts_tablet_manager.cc:1397] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.004s
I20250905 02:21:47.924252 2685 tablet_bootstrap.cc:492] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c: Bootstrap starting.
I20250905 02:21:47.925803 2688 tablet_bootstrap.cc:492] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187: Bootstrap starting.
I20250905 02:21:47.926056 2686 raft_consensus.cc:357] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:47.926931 2686 raft_consensus.cc:383] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:47.927289 2686 raft_consensus.cc:738] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe9e2ddd60246a7b7b650181e1df954, State: Initialized, Role: FOLLOWER
I20250905 02:21:47.928143 2686 consensus_queue.cc:260] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:47.930826 2685 tablet_bootstrap.cc:654] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:47.932072 2688 tablet_bootstrap.cc:654] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187: Neither blocks nor log segments found. Creating new log.
I20250905 02:21:47.938036 2686 ts_tablet_manager.cc:1428] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954: Time spent starting tablet: real 0.014s user 0.006s sys 0.003s
I20250905 02:21:47.942552 2688 tablet_bootstrap.cc:492] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187: No bootstrap required, opened a new log
I20250905 02:21:47.943135 2688 ts_tablet_manager.cc:1397] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20250905 02:21:47.949221 2688 raft_consensus.cc:357] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:47.950176 2688 raft_consensus.cc:383] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:47.950539 2688 raft_consensus.cc:738] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a403bb917c4ffa8f4be308c1dea187, State: Initialized, Role: FOLLOWER
I20250905 02:21:47.953222 2688 consensus_queue.cc:260] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:47.955387 2685 tablet_bootstrap.cc:492] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c: No bootstrap required, opened a new log
I20250905 02:21:47.955878 2685 ts_tablet_manager.cc:1397] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.018s
I20250905 02:21:47.956221 2688 ts_tablet_manager.cc:1428] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187: Time spent starting tablet: real 0.013s user 0.008s sys 0.001s
I20250905 02:21:47.958166 2685 raft_consensus.cc:357] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:47.958678 2685 raft_consensus.cc:383] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:21:47.958928 2685 raft_consensus.cc:738] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e2564f449540c8a2548e3cc047729c, State: Initialized, Role: FOLLOWER
I20250905 02:21:47.959520 2685 consensus_queue.cc:260] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:47.962230 2685 ts_tablet_manager.cc:1428] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20250905 02:21:48.023052 2693 raft_consensus.cc:491] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:21:48.023511 2693 raft_consensus.cc:513] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:48.025401 2693 leader_election.cc:290] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbe9e2ddd60246a7b7b650181e1df954 (127.31.48.131:33731), 04a403bb917c4ffa8f4be308c1dea187 (127.31.48.130:46163)
I20250905 02:21:48.038023 2639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5b0e019946d418ab8decde40e2e3771" candidate_uuid: "26e2564f449540c8a2548e3cc047729c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe9e2ddd60246a7b7b650181e1df954" is_pre_election: true
I20250905 02:21:48.038023 2565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5b0e019946d418ab8decde40e2e3771" candidate_uuid: "26e2564f449540c8a2548e3cc047729c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a403bb917c4ffa8f4be308c1dea187" is_pre_election: true
I20250905 02:21:48.039004 2639 raft_consensus.cc:2466] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e2564f449540c8a2548e3cc047729c in term 0.
I20250905 02:21:48.039038 2565 raft_consensus.cc:2466] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26e2564f449540c8a2548e3cc047729c in term 0.
I20250905 02:21:48.040407 2453 leader_election.cc:304] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e2564f449540c8a2548e3cc047729c, fbe9e2ddd60246a7b7b650181e1df954; no voters:
I20250905 02:21:48.041018 2693 raft_consensus.cc:2802] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:21:48.041276 2693 raft_consensus.cc:491] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:21:48.041488 2693 raft_consensus.cc:3058] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:48.046088 2693 raft_consensus.cc:513] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:48.047591 2693 leader_election.cc:290] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [CANDIDATE]: Term 1 election: Requested vote from peers fbe9e2ddd60246a7b7b650181e1df954 (127.31.48.131:33731), 04a403bb917c4ffa8f4be308c1dea187 (127.31.48.130:46163)
I20250905 02:21:48.048430 2639 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5b0e019946d418ab8decde40e2e3771" candidate_uuid: "26e2564f449540c8a2548e3cc047729c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe9e2ddd60246a7b7b650181e1df954"
I20250905 02:21:48.048590 2565 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5b0e019946d418ab8decde40e2e3771" candidate_uuid: "26e2564f449540c8a2548e3cc047729c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04a403bb917c4ffa8f4be308c1dea187"
I20250905 02:21:48.049010 2639 raft_consensus.cc:3058] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:48.049068 2565 raft_consensus.cc:3058] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:21:48.056435 2565 raft_consensus.cc:2466] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e2564f449540c8a2548e3cc047729c in term 1.
I20250905 02:21:48.056437 2639 raft_consensus.cc:2466] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26e2564f449540c8a2548e3cc047729c in term 1.
I20250905 02:21:48.057880 2454 leader_election.cc:304] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a403bb917c4ffa8f4be308c1dea187, 26e2564f449540c8a2548e3cc047729c; no voters:
I20250905 02:21:48.058583 2693 raft_consensus.cc:2802] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:21:48.059630 2693 raft_consensus.cc:695] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 1 LEADER]: Becoming Leader. State: Replica: 26e2564f449540c8a2548e3cc047729c, State: Running, Role: LEADER
I20250905 02:21:48.060256 2693 consensus_queue.cc:237] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } }
I20250905 02:21:48.066846 2387 catalog_manager.cc:5582] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c reported cstate change: term changed from 0 to 1, leader changed from <none> to 26e2564f449540c8a2548e3cc047729c (127.31.48.129). New cstate: current_term: 1 leader_uuid: "26e2564f449540c8a2548e3cc047729c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e2564f449540c8a2548e3cc047729c" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 46807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 } health_report { overall_health: UNKNOWN } } }
I20250905 02:21:48.404008 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:48.407213 2441 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:48.408591 2441 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:48.408890 2441 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:48.501283 2693 consensus_queue.cc:1035] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe9e2ddd60246a7b7b650181e1df954" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:48.513167 2696 consensus_queue.cc:1035] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04a403bb917c4ffa8f4be308c1dea187" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:21:49.412429 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:50.409808 2441 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:50.411067 2441 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:50.411383 2441 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:50.420082 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:51.428047 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:52.412549 2441 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:52.413843 2441 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:52.414156 2441 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:52.435580 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:53.442883 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:54.415010 2441 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:54.416322 2441 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:54.416671 2441 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:54.450456 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:55.457892 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:56.660890 2441 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:56.662518 2441 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:56.663837 2441 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:56.664671 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:57.674386 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
W20250905 02:21:57.687083 2660 debug-util.cc:398] Leaking SignalData structure 0x7b08001e5d80 after lost signal to thread 31941
W20250905 02:21:57.688079 2660 debug-util.cc:398] Leaking SignalData structure 0x7b08000497a0 after lost signal to thread 2422
W20250905 02:21:57.688992 2660 debug-util.cc:398] Leaking SignalData structure 0x7b08001cbba0 after lost signal to thread 2514
W20250905 02:21:57.689781 2660 debug-util.cc:398] Leaking SignalData structure 0x7b080010d3a0 after lost signal to thread 2589
W20250905 02:21:57.690544 2660 debug-util.cc:398] Leaking SignalData structure 0x7b080017a940 after lost signal to thread 2663
I20250905 02:21:58.664790 2441 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:21:58.666004 2441 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:21:58.666293 2441 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:21:58.680289 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:59.525589 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:21:59.555055 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:59.556021 31938 tablet_replica.cc:331] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c: stopping tablet replica
I20250905 02:21:59.556844 31938 raft_consensus.cc:2241] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:21:59.558138 31938 raft_consensus.cc:2270] T e5b0e019946d418ab8decde40e2e3771 P 26e2564f449540c8a2548e3cc047729c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:59.580143 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:21:59.592757 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:21:59.622892 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:59.623759 31938 tablet_replica.cc:331] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187: stopping tablet replica
I20250905 02:21:59.624596 31938 raft_consensus.cc:2241] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:59.625346 31938 raft_consensus.cc:2270] T e5b0e019946d418ab8decde40e2e3771 P 04a403bb917c4ffa8f4be308c1dea187 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:21:59.647979 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:21:59.664746 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:21:59.688293 2436 placement_policy_util.cc:407] tablet e5b0e019946d418ab8decde40e2e3771: detected majority of replicas (2 of 3) at location L0
I20250905 02:21:59.692010 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:21:59.692804 31938 tablet_replica.cc:331] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954: stopping tablet replica
I20250905 02:21:59.693463 31938 raft_consensus.cc:2241] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:21:59.693985 31938 raft_consensus.cc:2270] T e5b0e019946d418ab8decde40e2e3771 P fbe9e2ddd60246a7b7b650181e1df954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:00.242936 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:22:00.252775 31938 master.cc:561] Master@127.31.48.190:39685 shutting down...
I20250905 02:22:00.270042 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:00.270689 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:00.270973 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P abdec3ca9713415d9367a94e4f87e76b: stopping tablet replica
I20250905 02:22:00.289470 31938 master.cc:583] Master@127.31.48.190:39685 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotFixPlacementPolicy (14047 ms)
[ RUN ] AutoRebalancerTest.TestMaxMovesPerServer
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/auto_rebalancer-test.cc:516: Skipped
test is skipped; set KUDU_ALLOW_SLOW_TESTS=1 to run
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer (6 ms)
[ RUN ] AutoRebalancerTest.AutoRebalancingUnstableCluster
I20250905 02:22:00.328294 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:36341
I20250905 02:22:00.329438 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:00.334717 2712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.334792 2713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.335044 2715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:00.336382 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:00.337077 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:00.337255 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:00.337395 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038920337382 us; error 0 us; skew 500 ppm
I20250905 02:22:00.337808 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:00.339931 31938 webserver.cc:480] Webserver started at http://127.31.48.190:46339/ using document root <none> and password file <none>
I20250905 02:22:00.340312 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:00.340504 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:00.340708 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:00.341689 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "ccce443947d64ebe8a475a4b8bc96858"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.345567 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250905 02:22:00.348558 2720 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.349258 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:22:00.349510 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "ccce443947d64ebe8a475a4b8bc96858"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.349764 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:00.361389 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:00.362532 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:00.397681 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:36341
I20250905 02:22:00.397768 2771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:36341 every 8 connection(s)
I20250905 02:22:00.401662 2772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:00.411818 2772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858: Bootstrap starting.
I20250905 02:22:00.415879 2772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:00.419780 2772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858: No bootstrap required, opened a new log
I20250905 02:22:00.421658 2772 raft_consensus.cc:357] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccce443947d64ebe8a475a4b8bc96858" member_type: VOTER }
I20250905 02:22:00.422017 2772 raft_consensus.cc:383] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:00.422243 2772 raft_consensus.cc:738] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccce443947d64ebe8a475a4b8bc96858, State: Initialized, Role: FOLLOWER
I20250905 02:22:00.422813 2772 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccce443947d64ebe8a475a4b8bc96858" member_type: VOTER }
I20250905 02:22:00.423271 2772 raft_consensus.cc:397] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:00.423486 2772 raft_consensus.cc:491] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:00.423749 2772 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:00.427901 2772 raft_consensus.cc:513] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccce443947d64ebe8a475a4b8bc96858" member_type: VOTER }
I20250905 02:22:00.428436 2772 leader_election.cc:304] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccce443947d64ebe8a475a4b8bc96858; no voters:
I20250905 02:22:00.429461 2772 leader_election.cc:290] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:00.429780 2775 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:00.431228 2775 raft_consensus.cc:695] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 1 LEADER]: Becoming Leader. State: Replica: ccce443947d64ebe8a475a4b8bc96858, State: Running, Role: LEADER
I20250905 02:22:00.431947 2775 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccce443947d64ebe8a475a4b8bc96858" member_type: VOTER }
I20250905 02:22:00.432626 2772 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:00.436964 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccce443947d64ebe8a475a4b8bc96858. Latest consensus state: current_term: 1 leader_uuid: "ccce443947d64ebe8a475a4b8bc96858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccce443947d64ebe8a475a4b8bc96858" member_type: VOTER } }
I20250905 02:22:00.437676 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:00.437772 2776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccce443947d64ebe8a475a4b8bc96858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccce443947d64ebe8a475a4b8bc96858" member_type: VOTER } }
I20250905 02:22:00.438469 2776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:00.439739 2784 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:00.444331 2784 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:00.446763 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:22:00.452723 2784 catalog_manager.cc:1349] Generated new cluster ID: 3e2ed180a95e46668c76a7ee2139b9a5
I20250905 02:22:00.452932 2784 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:00.475843 2784 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:00.477110 2784 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:00.490811 2784 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858: Generated new TSK 0
I20250905 02:22:00.491358 2784 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:00.513679 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:00.518976 2793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.520239 2794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.521600 2796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:00.521865 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:00.522763 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:00.522943 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:00.523095 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038920523078 us; error 0 us; skew 500 ppm
I20250905 02:22:00.523548 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:00.525782 31938 webserver.cc:480] Webserver started at http://127.31.48.129:35937/ using document root <none> and password file <none>
I20250905 02:22:00.526239 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:00.526409 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:00.526651 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:00.527717 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "e14af3fd13ac47649ce20cae3651491d"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.532071 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 02:22:00.535269 2801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.535959 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:22:00.536226 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "e14af3fd13ac47649ce20cae3651491d"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.536527 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:00.562994 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:00.564178 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:00.565573 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:00.567751 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:00.567940 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.568202 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:00.568352 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.606591 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:42565
I20250905 02:22:00.606709 2863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:42565 every 8 connection(s)
I20250905 02:22:00.611057 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:00.618772 2868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:00.621666 31938 server_base.cc:1047] running on GCE node
W20250905 02:22:00.621784 2869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.623598 2871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:00.624547 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 02:22:00.624598 2864 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36341
W20250905 02:22:00.624791 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:00.624989 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038920624972 us; error 0 us; skew 500 ppm
I20250905 02:22:00.625025 2864 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:00.625504 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:00.625756 2864 heartbeater.cc:507] Master 127.31.48.190:36341 requested a full tablet report, sending...
I20250905 02:22:00.627991 2737 ts_manager.cc:194] Registered new tserver with Master: e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565)
I20250905 02:22:00.628010 31938 webserver.cc:480] Webserver started at http://127.31.48.130:43161/ using document root <none> and password file <none>
I20250905 02:22:00.628754 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:00.628950 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:00.629204 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:00.629904 2737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50022
I20250905 02:22:00.630304 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.634699 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:22:00.637738 2876 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.638515 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 02:22:00.638775 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.639042 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:00.651423 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:00.652622 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:00.654111 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:00.656275 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:00.656517 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.656733 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:00.656884 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.695989 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:32981
I20250905 02:22:00.696131 2938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:32981 every 8 connection(s)
I20250905 02:22:00.700764 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:00.706795 2942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.707870 2943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:00.711056 2945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:00.711591 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:00.712566 2939 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36341
I20250905 02:22:00.712579 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:00.712971 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:00.713016 2939 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:00.713177 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038920713165 us; error 0 us; skew 500 ppm
I20250905 02:22:00.713759 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:00.713770 2939 heartbeater.cc:507] Master 127.31.48.190:36341 requested a full tablet report, sending...
I20250905 02:22:00.715835 2737 ts_manager.cc:194] Registered new tserver with Master: 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:00.716337 31938 webserver.cc:480] Webserver started at http://127.31.48.131:45245/ using document root <none> and password file <none>
I20250905 02:22:00.716871 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:00.717056 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:00.717303 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:00.717367 2737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50030
I20250905 02:22:00.718600 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "7d12f406bc4148ba854b41a9cccc2197"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.722875 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20250905 02:22:00.725982 2950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.726636 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 02:22:00.726919 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "7d12f406bc4148ba854b41a9cccc2197"
format_stamp: "Formatted at 2025-09-05 02:22:00 on dist-test-slave-q32q"
I20250905 02:22:00.727198 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:00.736986 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:00.738103 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:00.739403 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:00.741566 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:00.741799 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.742017 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:00.742174 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:00.780805 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:34231
I20250905 02:22:00.780925 3012 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:34231 every 8 connection(s)
I20250905 02:22:00.794991 3013 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36341
I20250905 02:22:00.795394 3013 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:00.796111 3013 heartbeater.cc:507] Master 127.31.48.190:36341 requested a full tablet report, sending...
I20250905 02:22:00.797979 2737 ts_manager.cc:194] Registered new tserver with Master: 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:00.798441 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014625337s
I20250905 02:22:00.799752 2737 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50032
I20250905 02:22:01.632539 2864 heartbeater.cc:499] Master 127.31.48.190:36341 was elected leader, sending a full tablet report...
I20250905 02:22:01.719940 2939 heartbeater.cc:499] Master 127.31.48.190:36341 was elected leader, sending a full tablet report...
I20250905 02:22:01.802899 3013 heartbeater.cc:499] Master 127.31.48.190:36341 was elected leader, sending a full tablet report...
I20250905 02:22:01.831980 31938 test_util.cc:276] Using random seed: -2134190644
I20250905 02:22:01.858312 2737 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50040:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\252\252\252*\004\001\000TUUU""\004\001\000\252\252\252*\004\001\000TUUU"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:22:01.860706 2737 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:22:01.916086 2978 tablet_service.cc:1468] Processing CreateTablet for tablet 9330b5a26c9b4b3f920e33058c496be0 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250905 02:22:01.916416 2977 tablet_service.cc:1468] Processing CreateTablet for tablet a26e4c64be9d4ecb8536269990b65642 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250905 02:22:01.916887 2976 tablet_service.cc:1468] Processing CreateTablet for tablet 36694c39cdba46d9bdd426065c9426d4 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250905 02:22:01.918210 2978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9330b5a26c9b4b3f920e33058c496be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.919068 2827 tablet_service.cc:1468] Processing CreateTablet for tablet 36694c39cdba46d9bdd426065c9426d4 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250905 02:22:01.919498 2828 tablet_service.cc:1468] Processing CreateTablet for tablet a26e4c64be9d4ecb8536269990b65642 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250905 02:22:01.920540 2827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36694c39cdba46d9bdd426065c9426d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.921550 2976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36694c39cdba46d9bdd426065c9426d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.921458 2829 tablet_service.cc:1468] Processing CreateTablet for tablet 9330b5a26c9b4b3f920e33058c496be0 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250905 02:22:01.922693 2829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9330b5a26c9b4b3f920e33058c496be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.924573 2977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26e4c64be9d4ecb8536269990b65642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.926584 2828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26e4c64be9d4ecb8536269990b65642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.946818 3034 tablet_bootstrap.cc:492] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197: Bootstrap starting.
I20250905 02:22:01.949471 3035 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: Bootstrap starting.
I20250905 02:22:01.956753 3034 tablet_bootstrap.cc:654] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:01.957098 3035 tablet_bootstrap.cc:654] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:01.961826 2904 tablet_service.cc:1468] Processing CreateTablet for tablet 9330b5a26c9b4b3f920e33058c496be0 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION VALUES < 715827882
I20250905 02:22:01.963513 2904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9330b5a26c9b4b3f920e33058c496be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.964241 2902 tablet_service.cc:1468] Processing CreateTablet for tablet 36694c39cdba46d9bdd426065c9426d4 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION 1431655764 <= VALUES
I20250905 02:22:01.965582 2902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36694c39cdba46d9bdd426065c9426d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.965723 2903 tablet_service.cc:1468] Processing CreateTablet for tablet a26e4c64be9d4ecb8536269990b65642 (DEFAULT_TABLE table=test-workload [id=0f6db5dfee9840a492cd5cc68c931965]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 1431655764
I20250905 02:22:01.967046 2903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26e4c64be9d4ecb8536269990b65642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:01.968183 3035 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: No bootstrap required, opened a new log
I20250905 02:22:01.968708 3035 ts_tablet_manager.cc:1397] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.003s
I20250905 02:22:01.971735 3035 raft_consensus.cc:357] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:01.972711 3035 raft_consensus.cc:383] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:01.973059 3035 raft_consensus.cc:738] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Initialized, Role: FOLLOWER
I20250905 02:22:01.973941 3035 consensus_queue.cc:260] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:01.983916 3038 tablet_bootstrap.cc:492] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc: Bootstrap starting.
I20250905 02:22:01.986250 3035 ts_tablet_manager.cc:1428] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: Time spent starting tablet: real 0.017s user 0.009s sys 0.004s
I20250905 02:22:01.987397 3035 tablet_bootstrap.cc:492] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: Bootstrap starting.
I20250905 02:22:01.991931 3034 tablet_bootstrap.cc:492] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197: No bootstrap required, opened a new log
I20250905 02:22:01.992599 3034 ts_tablet_manager.cc:1397] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.009s
I20250905 02:22:01.993714 3035 tablet_bootstrap.cc:654] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:01.994470 3038 tablet_bootstrap.cc:654] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:01.995512 3034 raft_consensus.cc:357] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:01.996302 3034 raft_consensus.cc:383] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:01.996712 3034 raft_consensus.cc:738] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d12f406bc4148ba854b41a9cccc2197, State: Initialized, Role: FOLLOWER
I20250905 02:22:01.997818 3034 consensus_queue.cc:260] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.003274 3038 tablet_bootstrap.cc:492] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc: No bootstrap required, opened a new log
I20250905 02:22:02.003569 3035 tablet_bootstrap.cc:492] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: No bootstrap required, opened a new log
I20250905 02:22:02.003851 3038 ts_tablet_manager.cc:1397] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20250905 02:22:02.004184 3035 ts_tablet_manager.cc:1397] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20250905 02:22:02.003263 3034 ts_tablet_manager.cc:1428] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20250905 02:22:02.006983 3034 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197: Bootstrap starting.
I20250905 02:22:02.007009 3038 raft_consensus.cc:357] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.007416 3035 raft_consensus.cc:357] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.008105 3038 raft_consensus.cc:383] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.008273 3035 raft_consensus.cc:383] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.008560 3038 raft_consensus.cc:738] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bbb6a5eec54a85b6179da4b8da31fc, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.008740 3035 raft_consensus.cc:738] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.009814 3038 consensus_queue.cc:260] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.009825 3035 consensus_queue.cc:260] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.012449 3035 ts_tablet_manager.cc:1428] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20250905 02:22:02.013547 3035 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: Bootstrap starting.
I20250905 02:22:02.016911 3034 tablet_bootstrap.cc:654] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:02.019493 3038 ts_tablet_manager.cc:1428] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc: Time spent starting tablet: real 0.015s user 0.010s sys 0.003s
I20250905 02:22:02.020572 3035 tablet_bootstrap.cc:654] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:02.026208 3038 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: Bootstrap starting.
I20250905 02:22:02.030278 3034 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197: No bootstrap required, opened a new log
I20250905 02:22:02.030857 3034 ts_tablet_manager.cc:1397] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.005s
I20250905 02:22:02.033684 3038 tablet_bootstrap.cc:654] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:02.034009 3034 raft_consensus.cc:357] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.034873 3034 raft_consensus.cc:383] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.034965 3035 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: No bootstrap required, opened a new log
I20250905 02:22:02.035840 3035 ts_tablet_manager.cc:1397] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: Time spent bootstrapping tablet: real 0.023s user 0.015s sys 0.004s
I20250905 02:22:02.035318 3034 raft_consensus.cc:738] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d12f406bc4148ba854b41a9cccc2197, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.036818 3034 consensus_queue.cc:260] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.039036 3035 raft_consensus.cc:357] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.039480 3034 ts_tablet_manager.cc:1428] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20250905 02:22:02.039848 3035 raft_consensus.cc:383] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.040041 3038 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: No bootstrap required, opened a new log
I20250905 02:22:02.040251 3035 raft_consensus.cc:738] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.040668 3038 ts_tablet_manager.cc:1397] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20250905 02:22:02.040840 3034 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: Bootstrap starting.
I20250905 02:22:02.041450 3035 consensus_queue.cc:260] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.043912 3035 ts_tablet_manager.cc:1428] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 02:22:02.043537 3038 raft_consensus.cc:357] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.044433 3038 raft_consensus.cc:383] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.044766 3038 raft_consensus.cc:738] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bbb6a5eec54a85b6179da4b8da31fc, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.045708 3038 consensus_queue.cc:260] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.047577 3034 tablet_bootstrap.cc:654] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:02.047999 3038 ts_tablet_manager.cc:1428] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250905 02:22:02.049031 3038 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc: Bootstrap starting.
I20250905 02:22:02.057152 3038 tablet_bootstrap.cc:654] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:02.060355 3034 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: No bootstrap required, opened a new log
I20250905 02:22:02.061017 3034 ts_tablet_manager.cc:1397] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.008s
I20250905 02:22:02.063105 3038 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc: No bootstrap required, opened a new log
I20250905 02:22:02.063634 3038 ts_tablet_manager.cc:1397] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20250905 02:22:02.064172 3034 raft_consensus.cc:357] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.064982 3034 raft_consensus.cc:383] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.065296 3034 raft_consensus.cc:738] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d12f406bc4148ba854b41a9cccc2197, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.066031 3034 consensus_queue.cc:260] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.066507 3038 raft_consensus.cc:357] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.067193 3038 raft_consensus.cc:383] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:02.067507 3038 raft_consensus.cc:738] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27bbb6a5eec54a85b6179da4b8da31fc, State: Initialized, Role: FOLLOWER
I20250905 02:22:02.068526 3034 ts_tablet_manager.cc:1428] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250905 02:22:02.068282 3038 consensus_queue.cc:260] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.070304 3038 ts_tablet_manager.cc:1428] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250905 02:22:02.076464 3042 raft_consensus.cc:491] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:02.077093 3042 raft_consensus.cc:513] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.081542 3042 leader_election.cc:290] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:02.091415 2839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d" is_pre_election: true
I20250905 02:22:02.091416 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" is_pre_election: true
I20250905 02:22:02.092149 2839 raft_consensus.cc:2466] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 0.
I20250905 02:22:02.092228 2914 raft_consensus.cc:2466] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 0.
I20250905 02:22:02.093330 2953 leader_election.cc:304] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:02.093979 3042 raft_consensus.cc:2802] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:02.094305 3042 raft_consensus.cc:491] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:02.094547 3042 raft_consensus.cc:3058] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.099278 3042 raft_consensus.cc:513] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.101264 3042 leader_election.cc:290] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 1 election: Requested vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:02.101716 2839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
I20250905 02:22:02.101966 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
I20250905 02:22:02.102357 2839 raft_consensus.cc:3058] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.102525 2914 raft_consensus.cc:3058] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.107542 2914 raft_consensus.cc:2466] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 1.
I20250905 02:22:02.107545 2839 raft_consensus.cc:2466] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 1.
I20250905 02:22:02.108830 2951 leader_election.cc:304] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, 7d12f406bc4148ba854b41a9cccc2197; no voters:
I20250905 02:22:02.109478 3042 raft_consensus.cc:2802] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:02.110571 3042 raft_consensus.cc:695] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Becoming Leader. State: Replica: 7d12f406bc4148ba854b41a9cccc2197, State: Running, Role: LEADER
I20250905 02:22:02.111229 3042 consensus_queue.cc:237] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.119196 3037 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:02.119639 3037 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.120149 2735 catalog_manager.cc:5582] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131). New cstate: current_term: 1 leader_uuid: "7d12f406bc4148ba854b41a9cccc2197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:02.121963 3037 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:02.134275 2988 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:02.134864 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" is_pre_election: true
I20250905 02:22:02.135002 2988 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 0.
I20250905 02:22:02.135555 2914 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 0.
I20250905 02:22:02.136293 2802 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:02.137009 3037 raft_consensus.cc:2802] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:02.137282 3037 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:02.137539 3037 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.142280 3037 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.143630 3037 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 election: Requested vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:02.144532 2988 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:02.144629 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
I20250905 02:22:02.144992 2988 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.145095 2914 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.147221 3037 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:02.147735 3037 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.149674 3037 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981), 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:02.150827 2913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" is_pre_election: true
I20250905 02:22:02.150991 2914 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 1.
I20250905 02:22:02.151254 2988 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 1.
I20250905 02:22:02.151511 2913 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 0.
I20250905 02:22:02.151566 2987 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:02.152303 2987 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 0.
I20250905 02:22:02.152411 2802 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:02.153169 3037 raft_consensus.cc:2802] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:02.153553 2802 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:02.154414 3037 raft_consensus.cc:695] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Becoming Leader. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Running, Role: LEADER
I20250905 02:22:02.155349 3056 raft_consensus.cc:2802] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:02.155789 3056 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:02.155272 3037 consensus_queue.cc:237] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:02.156152 3056 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.162079 3056 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.162915 2735 catalog_manager.cc:5582] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d reported cstate change: term changed from 0 to 1, leader changed from <none> to e14af3fd13ac47649ce20cae3651491d (127.31.48.129). New cstate: current_term: 1 leader_uuid: "e14af3fd13ac47649ce20cae3651491d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:02.163602 3056 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 election: Requested vote from peers 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981), 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:02.164335 2913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
I20250905 02:22:02.164623 2987 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:02.164918 2913 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.165109 2987 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:02.170027 2913 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 1.
I20250905 02:22:02.170200 2987 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 1.
I20250905 02:22:02.170982 2802 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:02.171597 3056 raft_consensus.cc:2802] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:02.172044 3056 raft_consensus.cc:695] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Becoming Leader. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Running, Role: LEADER
I20250905 02:22:02.172856 3056 consensus_queue.cc:237] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } }
I20250905 02:22:02.180442 2735 catalog_manager.cc:5582] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d reported cstate change: term changed from 0 to 1, leader changed from <none> to e14af3fd13ac47649ce20cae3651491d (127.31.48.129). New cstate: current_term: 1 leader_uuid: "e14af3fd13ac47649ce20cae3651491d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:02.209471 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:02.216152 3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:02.217407 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:02.219928 31938 server_base.cc:1047] running on GCE node
W20250905 02:22:02.220000 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:02.221611 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:02.221865 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:02.222047 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038922222028 us; error 0 us; skew 500 ppm
I20250905 02:22:02.222546 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:02.225402 31938 webserver.cc:480] Webserver started at http://127.31.48.132:41575/ using document root <none> and password file <none>
I20250905 02:22:02.225909 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:02.226117 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:02.226423 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:02.227644 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "2cb93e1290274d558112fb1c48347cba"
format_stamp: "Formatted at 2025-09-05 02:22:02 on dist-test-slave-q32q"
I20250905 02:22:02.232472 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 02:22:02.235741 3067 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:02.236511 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:22:02.236790 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "2cb93e1290274d558112fb1c48347cba"
format_stamp: "Formatted at 2025-09-05 02:22:02 on dist-test-slave-q32q"
I20250905 02:22:02.237054 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:02.254913 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:02.256273 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:02.257858 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:02.260241 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:02.260481 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250905 02:22:02.260730 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:02.260887 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:02.301263 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:38077
I20250905 02:22:02.301379 3129 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:38077 every 8 connection(s)
I20250905 02:22:02.314641 3130 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36341
I20250905 02:22:02.314987 3130 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:02.315686 3130 heartbeater.cc:507] Master 127.31.48.190:36341 requested a full tablet report, sending...
I20250905 02:22:02.317427 2735 ts_manager.cc:194] Registered new tserver with Master: 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077)
I20250905 02:22:02.318830 2735 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50052
I20250905 02:22:02.368265 3042 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:22:02.375617 3051 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250905 02:22:02.375931 3051 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250905 02:22:02.408875 3056 consensus_queue.cc:1035] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:02.414572 3056 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:22:02.417640 3137 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:22:02.446597 2790 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:02.463349 2839 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0"
dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
mode: GRACEFUL
new_leader_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
from {username='slave'} at 127.0.0.1:42468
I20250905 02:22:02.464282 2839 raft_consensus.cc:604] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Received request to transfer leadership to TS 27bbb6a5eec54a85b6179da4b8da31fc
I20250905 02:22:02.464208 2988 consensus_queue.cc:237] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } }
I20250905 02:22:02.465616 2790 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250905 02:22:02.466034 2790 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:02.466241 3056 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:02.471096 2839 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:22:02.472563 2914 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:22:02.473508 3137 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:02.474227 3042 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:02.482016 2914 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:02.491971 2839 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
W20250905 02:22:02.495703 2952 consensus_peers.cc:489] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 -> Peer 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077): Couldn't send request to peer 2cb93e1290274d558112fb1c48347cba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a26e4c64be9d4ecb8536269990b65642. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:02.499316 2735 catalog_manager.cc:5582] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc reported cstate change: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "7d12f406bc4148ba854b41a9cccc2197" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:02.505308 3056 consensus_queue.cc:1035] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:02.688428 3051 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:02.706473 2839 consensus_queue.cc:237] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } }
I20250905 02:22:02.711920 2988 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:22:02.713514 2914 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:22:02.713418 3056 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:02.714893 3148 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:22:02.721978 2803 consensus_peers.cc:489] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d -> Peer 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077): Couldn't send request to peer 2cb93e1290274d558112fb1c48347cba. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36694c39cdba46d9bdd426065c9426d4. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:02.723647 2914 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:02.723845 2988 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:02.734009 2737 catalog_manager.cc:5582] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc reported cstate change: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "e14af3fd13ac47649ce20cae3651491d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:02.876164 3150 ts_tablet_manager.cc:927] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: Initiating tablet copy from peer 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:02.902046 3150 tablet_copy_client.cc:323] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.131:34231
I20250905 02:22:02.928535 2998 tablet_copy_service.cc:140] P 7d12f406bc4148ba854b41a9cccc2197: Received BeginTabletCopySession request for tablet a26e4c64be9d4ecb8536269990b65642 from peer 2cb93e1290274d558112fb1c48347cba ({username='slave'} at 127.0.0.1:39448)
I20250905 02:22:02.929365 2998 tablet_copy_service.cc:161] P 7d12f406bc4148ba854b41a9cccc2197: Beginning new tablet copy session on tablet a26e4c64be9d4ecb8536269990b65642 from peer 2cb93e1290274d558112fb1c48347cba at {username='slave'} at 127.0.0.1:39448: session id = 2cb93e1290274d558112fb1c48347cba-a26e4c64be9d4ecb8536269990b65642
I20250905 02:22:03.133162 3037 raft_consensus.cc:991] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: : Instructing follower 27bbb6a5eec54a85b6179da4b8da31fc to start an election
I20250905 02:22:03.134013 3037 raft_consensus.cc:1079] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Signalling peer 27bbb6a5eec54a85b6179da4b8da31fc to start an election
I20250905 02:22:03.248899 3056 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 2cb93e1290274d558112fb1c48347cba (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } } }
I20250905 02:22:03.321302 3130 heartbeater.cc:499] Master 127.31.48.190:36341 was elected leader, sending a full tablet report...
I20250905 02:22:03.526042 2998 tablet_copy_source_session.cc:215] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:03.535869 2913 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0"
dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
from {username='slave'} at 127.0.0.1:38898
I20250905 02:22:03.536613 2913 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 02:22:03.537055 2913 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:03.542713 2913 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:03.546905 3150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a26e4c64be9d4ecb8536269990b65642. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:03.556326 2913 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 2 election: Requested vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565)
I20250905 02:22:03.569633 2914 raft_consensus.cc:1238] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Rejecting Update request from peer e14af3fd13ac47649ce20cae3651491d for earlier term 1. Current term is 2. Ops: []
I20250905 02:22:03.571216 3055 consensus_queue.cc:1046] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:03.574215 3150 tablet_copy_client.cc:806] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:03.574970 3150 tablet_copy_client.cc:670] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:03.578068 3157 ts_tablet_manager.cc:927] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: Initiating tablet copy from peer e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565)
I20250905 02:22:03.589735 2988 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:03.590224 2988 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:03.596309 2988 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27bbb6a5eec54a85b6179da4b8da31fc in term 2.
I20250905 02:22:03.597616 2877 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, 7d12f406bc4148ba854b41a9cccc2197; no voters:
I20250905 02:22:03.599761 3157 tablet_copy_client.cc:323] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.129:42565
I20250905 02:22:03.600458 3160 raft_consensus.cc:2802] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:03.602936 3150 tablet_copy_client.cc:538] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:03.614180 3150 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: Bootstrap starting.
I20250905 02:22:03.629256 3160 raft_consensus.cc:695] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 LEADER]: Becoming Leader. State: Replica: 27bbb6a5eec54a85b6179da4b8da31fc, State: Running, Role: LEADER
I20250905 02:22:03.630221 3160 consensus_queue.cc:237] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:03.630738 2839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
I20250905 02:22:03.631337 2839 raft_consensus.cc:3053] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Stepping down as leader of term 1
I20250905 02:22:03.631696 2839 raft_consensus.cc:738] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Running, Role: LEADER
I20250905 02:22:03.637429 2849 tablet_copy_service.cc:140] P e14af3fd13ac47649ce20cae3651491d: Received BeginTabletCopySession request for tablet 36694c39cdba46d9bdd426065c9426d4 from peer 2cb93e1290274d558112fb1c48347cba ({username='slave'} at 127.0.0.1:42486)
I20250905 02:22:03.638017 2849 tablet_copy_service.cc:161] P e14af3fd13ac47649ce20cae3651491d: Beginning new tablet copy session on tablet 36694c39cdba46d9bdd426065c9426d4 from peer 2cb93e1290274d558112fb1c48347cba at {username='slave'} at 127.0.0.1:42486: session id = 2cb93e1290274d558112fb1c48347cba-36694c39cdba46d9bdd426065c9426d4
I20250905 02:22:03.639158 2839 consensus_queue.cc:260] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:03.641068 2839 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:03.643755 3150 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:03.644714 3150 tablet_bootstrap.cc:492] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: Bootstrap complete.
I20250905 02:22:03.645452 3150 ts_tablet_manager.cc:1397] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.012s
I20250905 02:22:03.646131 2849 tablet_copy_source_session.cc:215] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:03.648298 3150 raft_consensus.cc:357] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } }
I20250905 02:22:03.649022 3150 raft_consensus.cc:738] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2cb93e1290274d558112fb1c48347cba, State: Initialized, Role: LEARNER
I20250905 02:22:03.649441 3157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36694c39cdba46d9bdd426065c9426d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:03.649530 3150 consensus_queue.cc:260] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } }
I20250905 02:22:03.652215 3150 ts_tablet_manager.cc:1428] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250905 02:22:03.656746 2998 tablet_copy_service.cc:342] P 7d12f406bc4148ba854b41a9cccc2197: Request end of tablet copy session 2cb93e1290274d558112fb1c48347cba-a26e4c64be9d4ecb8536269990b65642 received from {username='slave'} at 127.0.0.1:39448
I20250905 02:22:03.657214 2998 tablet_copy_service.cc:434] P 7d12f406bc4148ba854b41a9cccc2197: ending tablet copy session 2cb93e1290274d558112fb1c48347cba-a26e4c64be9d4ecb8536269990b65642 on tablet a26e4c64be9d4ecb8536269990b65642 with peer 2cb93e1290274d558112fb1c48347cba
I20250905 02:22:03.667426 3157 tablet_copy_client.cc:806] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:03.668061 3157 tablet_copy_client.cc:670] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:03.672152 3157 tablet_copy_client.cc:538] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:03.674697 2839 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27bbb6a5eec54a85b6179da4b8da31fc in term 2.
I20250905 02:22:03.681887 3157 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: Bootstrap starting.
I20250905 02:22:03.683441 2736 catalog_manager.cc:5582] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc reported cstate change: term changed from 1 to 2, leader changed from e14af3fd13ac47649ce20cae3651491d (127.31.48.129) to 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130). New cstate: current_term: 2 leader_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:03.702817 3157 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:03.703579 3157 tablet_bootstrap.cc:492] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: Bootstrap complete.
I20250905 02:22:03.704176 3157 ts_tablet_manager.cc:1397] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.002s
I20250905 02:22:03.706341 3157 raft_consensus.cc:357] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } }
I20250905 02:22:03.707024 3157 raft_consensus.cc:738] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2cb93e1290274d558112fb1c48347cba, State: Initialized, Role: LEARNER
I20250905 02:22:03.707604 3157 consensus_queue.cc:260] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: true } }
I20250905 02:22:03.709827 3157 ts_tablet_manager.cc:1428] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 02:22:03.711366 2849 tablet_copy_service.cc:342] P e14af3fd13ac47649ce20cae3651491d: Request end of tablet copy session 2cb93e1290274d558112fb1c48347cba-36694c39cdba46d9bdd426065c9426d4 received from {username='slave'} at 127.0.0.1:42486
I20250905 02:22:03.711783 2849 tablet_copy_service.cc:434] P e14af3fd13ac47649ce20cae3651491d: ending tablet copy session 2cb93e1290274d558112fb1c48347cba-36694c39cdba46d9bdd426065c9426d4 on tablet 36694c39cdba46d9bdd426065c9426d4 with peer 2cb93e1290274d558112fb1c48347cba
I20250905 02:22:03.728227 3105 raft_consensus.cc:1215] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 02:22:03.733841 3105 raft_consensus.cc:1215] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 02:22:03.889341 2988 raft_consensus.cc:1273] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Refusing update from remote peer 27bbb6a5eec54a85b6179da4b8da31fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:03.890554 3160 consensus_queue.cc:1035] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:03.897409 2839 raft_consensus.cc:1273] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Refusing update from remote peer 27bbb6a5eec54a85b6179da4b8da31fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:03.899366 3160 consensus_queue.cc:1035] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:04.100839 3056 raft_consensus.cc:1062] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: attempting to promote NON_VOTER 2cb93e1290274d558112fb1c48347cba to VOTER
I20250905 02:22:04.102896 3056 consensus_queue.cc:237] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:04.107637 2987 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:04.107753 3105 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:04.108314 2914 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:04.109174 3056 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:04.109982 3161 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:04.110960 3153 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:04.126093 2987 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.131726 2914 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.138501 2736 catalog_manager.cc:5582] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 reported cstate change: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e14af3fd13ac47649ce20cae3651491d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.171202 3136 raft_consensus.cc:1062] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: attempting to promote NON_VOTER 2cb93e1290274d558112fb1c48347cba to VOTER
I20250905 02:22:04.173408 3136 consensus_queue.cc:237] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:04.179620 3105 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 LEARNER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:04.180122 2914 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:04.181507 3170 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:22:04.183691 3182 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:22:04.184599 2839 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:04.186110 3170 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:04.195675 2839 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.196635 2914 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.206740 2737 catalog_manager.cc:5582] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d reported cstate change: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d12f406bc4148ba854b41a9cccc2197" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.273231 3182 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.281631 3105 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } attrs { replace: true } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.355696 3056 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.364439 3105 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 2cb93e1290274d558112fb1c48347cba (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } attrs { replace: true } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
W20250905 02:22:04.448534 2768 debug-util.cc:398] Leaking SignalData structure 0x7b0800117e60 after lost signal to thread 31941
W20250905 02:22:04.449445 2768 debug-util.cc:398] Leaking SignalData structure 0x7b0800098f20 after lost signal to thread 2771
W20250905 02:22:04.450835 2768 debug-util.cc:398] Leaking SignalData structure 0x7b080016cb60 after lost signal to thread 2938
W20250905 02:22:04.451789 2768 debug-util.cc:398] Leaking SignalData structure 0x7b08000b84e0 after lost signal to thread 3029
W20250905 02:22:04.452673 2768 debug-util.cc:398] Leaking SignalData structure 0x7b08001956a0 after lost signal to thread 3129
I20250905 02:22:04.467017 2790 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:04.470356 2790 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:04.470921 2790 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:04.577976 2988 consensus_queue.cc:237] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:04.584687 2914 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:04.585402 3105 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:04.586850 3182 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:04.587761 3136 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:04.599169 2914 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e14af3fd13ac47649ce20cae3651491d (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.599823 3105 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e14af3fd13ac47649ce20cae3651491d (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.608510 2838 consensus_queue.cc:237] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:04.614553 2736 catalog_manager.cc:5582] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba reported cstate change: config changed from index 3 to 4, VOTER e14af3fd13ac47649ce20cae3651491d (127.31.48.129) evicted. New cstate: current_term: 1 leader_uuid: "7d12f406bc4148ba854b41a9cccc2197" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.616811 2988 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:04.617238 3104 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Refusing update from remote peer e14af3fd13ac47649ce20cae3651491d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:04.618615 3189 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 02:22:04.621099 3189 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:04.627676 2987 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.629534 3104 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.641568 2737 catalog_manager.cc:5582] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 reported cstate change: config changed from index 3 to 4, VOTER 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130) evicted. New cstate: current_term: 1 leader_uuid: "e14af3fd13ac47649ce20cae3651491d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.801868 2828 tablet_service.cc:1515] Processing DeleteTablet for tablet a26e4c64be9d4ecb8536269990b65642 with delete_type TABLET_DATA_TOMBSTONED (TS e14af3fd13ac47649ce20cae3651491d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42454
I20250905 02:22:04.804121 3190 tablet_replica.cc:331] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: stopping tablet replica
I20250905 02:22:04.805047 3190 raft_consensus.cc:2241] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:04.805855 3190 raft_consensus.cc:2270] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:04.809100 3190 ts_tablet_manager.cc:1905] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:22:04.825786 3190 ts_tablet_manager.cc:1918] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 02:22:04.826292 3190 log.cc:1199] T a26e4c64be9d4ecb8536269990b65642 P e14af3fd13ac47649ce20cae3651491d: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-0-root/wals/a26e4c64be9d4ecb8536269990b65642
I20250905 02:22:04.828591 2723 catalog_manager.cc:4928] TS e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565): tablet a26e4c64be9d4ecb8536269990b65642 (table test-workload [id=0f6db5dfee9840a492cd5cc68c931965]) successfully deleted
I20250905 02:22:04.910248 3153 raft_consensus.cc:2953] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:04.918124 2723 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 36694c39cdba46d9bdd426065c9426d4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:05.015614 2904 tablet_service.cc:1515] Processing DeleteTablet for tablet 36694c39cdba46d9bdd426065c9426d4 with delete_type TABLET_DATA_TOMBSTONED (TS 27bbb6a5eec54a85b6179da4b8da31fc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:38890
I20250905 02:22:05.017954 3191 tablet_replica.cc:331] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: stopping tablet replica
I20250905 02:22:05.018890 3191 raft_consensus.cc:2241] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:05.019598 3191 raft_consensus.cc:2270] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:05.023332 3191 ts_tablet_manager.cc:1905] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:22:05.040910 3191 ts_tablet_manager.cc:1918] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20250905 02:22:05.041225 3191 log.cc:1199] T 36694c39cdba46d9bdd426065c9426d4 P 27bbb6a5eec54a85b6179da4b8da31fc: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.AutoRebalancingUnstableCluster.1757038820745333-31938-0/minicluster-data/ts-1-root/wals/36694c39cdba46d9bdd426065c9426d4
I20250905 02:22:05.042822 2721 catalog_manager.cc:4928] TS 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981): tablet 36694c39cdba46d9bdd426065c9426d4 (table test-workload [id=0f6db5dfee9840a492cd5cc68c931965]) successfully deleted
I20250905 02:22:05.055979 3170 raft_consensus.cc:2953] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e14af3fd13ac47649ce20cae3651491d (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } } }
I20250905 02:22:05.063759 2721 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a26e4c64be9d4ecb8536269990b65642 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:06.471797 2790 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:06.474644 2790 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:06.475065 2790 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:07.732928 3198 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e14af3fd13ac47649ce20cae3651491d)
I20250905 02:22:07.733635 3198 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:07.736543 3195 raft_consensus.cc:491] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7d12f406bc4148ba854b41a9cccc2197)
I20250905 02:22:07.738202 3195 raft_consensus.cc:513] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:07.739997 2839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d" is_pre_election: true
I20250905 02:22:07.741700 3195 leader_election.cc:290] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077)
I20250905 02:22:07.737059 3198 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:07.747812 3194 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27bbb6a5eec54a85b6179da4b8da31fc)
I20250905 02:22:07.748080 3198 raft_consensus.cc:491] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7d12f406bc4148ba854b41a9cccc2197)
I20250905 02:22:07.751123 3194 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:07.754853 2988 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:07.751034 3198 raft_consensus.cc:513] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:07.763671 3197 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 27bbb6a5eec54a85b6179da4b8da31fc)
I20250905 02:22:07.764516 3198 leader_election.cc:290] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981), 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:07.764380 3197 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:07.767858 2988 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:07.768885 2988 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e14af3fd13ac47649ce20cae3651491d in term 2.
I20250905 02:22:07.770064 2802 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:07.770964 3153 raft_consensus.cc:2802] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:22:07.771247 3153 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 27bbb6a5eec54a85b6179da4b8da31fc)
I20250905 02:22:07.771580 3153 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:07.771392 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" is_pre_election: true
I20250905 02:22:07.771288 3136 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e14af3fd13ac47649ce20cae3651491d)
I20250905 02:22:07.773327 2985 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:07.773851 3136 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:07.775566 2985 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb93e1290274d558112fb1c48347cba in term 1.
I20250905 02:22:07.775851 3136 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077)
I20250905 02:22:07.778683 3104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2cb93e1290274d558112fb1c48347cba" is_pre_election: true
I20250905 02:22:07.782236 3104 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 1.
I20250905 02:22:07.783937 2952 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cb93e1290274d558112fb1c48347cba, 7d12f406bc4148ba854b41a9cccc2197; no voters:
I20250905 02:22:07.766523 3197 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
W20250905 02:22:07.794163 2803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.010s sys 0.002s
W20250905 02:22:07.794672 2803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s user 0.010s sys 0.002s
I20250905 02:22:07.801131 3194 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:07.801147 3153 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:07.797037 3136 raft_consensus.cc:2802] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:07.802354 3136 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e14af3fd13ac47649ce20cae3651491d)
I20250905 02:22:07.802768 3136 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:07.804487 2986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:07.807113 3153 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 3 election: Requested vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981)
I20250905 02:22:07.807744 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "e14af3fd13ac47649ce20cae3651491d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
I20250905 02:22:07.809952 2802 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e14af3fd13ac47649ce20cae3651491d; no voters: 27bbb6a5eec54a85b6179da4b8da31fc, 7d12f406bc4148ba854b41a9cccc2197
I20250905 02:22:07.810374 2839 raft_consensus.cc:1238] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Rejecting Update request from peer 27bbb6a5eec54a85b6179da4b8da31fc for earlier term 2. Current term is 3. Ops: []
I20250905 02:22:07.812204 3177 consensus_queue.cc:1046] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:22:07.813540 2839 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d" is_pre_election: true
I20250905 02:22:07.814756 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d" is_pre_election: true
I20250905 02:22:07.815918 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" is_pre_election: true
I20250905 02:22:07.817763 3193 raft_consensus.cc:3053] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 LEADER]: Stepping down as leader of term 2
I20250905 02:22:07.818179 3193 raft_consensus.cc:738] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 27bbb6a5eec54a85b6179da4b8da31fc, State: Running, Role: LEADER
I20250905 02:22:07.812173 3153 raft_consensus.cc:2747] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250905 02:22:07.819178 3193 consensus_queue.cc:260] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:07.819880 2838 raft_consensus.cc:2391] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d12f406bc4148ba854b41a9cccc2197 in current term 3: Already voted for candidate e14af3fd13ac47649ce20cae3651491d in this term.
I20250905 02:22:07.820889 3193 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:07.841508 3136 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:07.846714 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
I20250905 02:22:07.847400 3068 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2cb93e1290274d558112fb1c48347cba, 7d12f406bc4148ba854b41a9cccc2197; no voters: e14af3fd13ac47649ce20cae3651491d
I20250905 02:22:07.849053 3199 raft_consensus.cc:2802] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:07.849167 2986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:07.849699 3199 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Starting leader election (detected failure of leader e14af3fd13ac47649ce20cae3651491d)
I20250905 02:22:07.850138 3199 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:07.852259 3136 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 2 election: Requested vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077)
I20250905 02:22:07.852667 3104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2cb93e1290274d558112fb1c48347cba"
I20250905 02:22:07.853765 2952 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197; no voters: 2cb93e1290274d558112fb1c48347cba, e14af3fd13ac47649ce20cae3651491d
I20250905 02:22:07.853765 2951 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197; no voters: 27bbb6a5eec54a85b6179da4b8da31fc, e14af3fd13ac47649ce20cae3651491d
I20250905 02:22:07.855486 3194 raft_consensus.cc:2747] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 02:22:07.855976 3136 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:07.858062 3199 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:07.860069 3199 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 election: Requested vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:07.860860 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
I20250905 02:22:07.861874 2986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:07.863467 3136 raft_consensus.cc:2747] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 02:22:07.863777 3104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2cb93e1290274d558112fb1c48347cba" is_pre_election: true
I20250905 02:22:07.865922 3068 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cb93e1290274d558112fb1c48347cba; no voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d
I20250905 02:22:07.867445 3199 raft_consensus.cc:2747] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250905 02:22:07.868906 2878 leader_election.cc:304] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc; no voters: 2cb93e1290274d558112fb1c48347cba, 7d12f406bc4148ba854b41a9cccc2197
I20250905 02:22:07.869925 3193 raft_consensus.cc:2747] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 02:22:07.873345 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" is_pre_election: true
I20250905 02:22:07.874810 3068 leader_election.cc:304] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cb93e1290274d558112fb1c48347cba; no voters: 27bbb6a5eec54a85b6179da4b8da31fc, 7d12f406bc4148ba854b41a9cccc2197
I20250905 02:22:07.875694 3199 raft_consensus.cc:2747] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 02:22:07.942888 2935 debug-util.cc:398] Leaking SignalData structure 0x7b080010f5a0 after lost signal to thread 31941
W20250905 02:22:07.943969 2935 debug-util.cc:398] Leaking SignalData structure 0x7b08000913e0 after lost signal to thread 2771
W20250905 02:22:07.945256 2935 debug-util.cc:398] Leaking SignalData structure 0x7b08001f1800 after lost signal to thread 2863
W20250905 02:22:07.946415 2935 debug-util.cc:398] Leaking SignalData structure 0x7b08001c8860 after lost signal to thread 2938
W20250905 02:22:07.947607 2935 debug-util.cc:398] Leaking SignalData structure 0x7b08001b48e0 after lost signal to thread 3012
W20250905 02:22:07.948858 2935 debug-util.cc:398] Leaking SignalData structure 0x7b080003e4e0 after lost signal to thread 3129
I20250905 02:22:08.098069 3104 raft_consensus.cc:1238] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Rejecting Update request from peer e14af3fd13ac47649ce20cae3651491d for earlier term 1. Current term is 2. Ops: []
I20250905 02:22:08.099334 3200 consensus_queue.cc:1046] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:22:08.106168 3153 raft_consensus.cc:3053] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Stepping down as leader of term 1
I20250905 02:22:08.106520 3153 raft_consensus.cc:738] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e14af3fd13ac47649ce20cae3651491d, State: Running, Role: LEADER
I20250905 02:22:08.107236 3153 consensus_queue.cc:260] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:08.108222 3153 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:08.476078 2790 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:08.480260 2790 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:08.480865 2790 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:08.713240 3205 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:08.713892 3205 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:08.716073 3205 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565)
I20250905 02:22:08.717236 2986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197" is_pre_election: true
I20250905 02:22:08.718005 2986 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27bbb6a5eec54a85b6179da4b8da31fc in term 3.
I20250905 02:22:08.717996 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d" is_pre_election: true
I20250905 02:22:08.718818 2838 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27bbb6a5eec54a85b6179da4b8da31fc in term 3.
I20250905 02:22:08.719445 2877 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, 7d12f406bc4148ba854b41a9cccc2197; no voters:
I20250905 02:22:08.720572 3205 raft_consensus.cc:2802] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 02:22:08.720959 3205 raft_consensus.cc:491] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:08.721300 3205 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 3 FOLLOWER]: Advancing to term 4
I20250905 02:22:08.728835 3205 raft_consensus.cc:513] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:08.732002 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
I20250905 02:22:08.731961 2986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9330b5a26c9b4b3f920e33058c496be0" candidate_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:08.732815 2838 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Advancing to term 4
I20250905 02:22:08.732950 2986 raft_consensus.cc:3058] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 FOLLOWER]: Advancing to term 4
I20250905 02:22:08.740180 2838 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27bbb6a5eec54a85b6179da4b8da31fc in term 4.
I20250905 02:22:08.741400 2879 leader_election.cc:304] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:08.746789 3205 leader_election.cc:290] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [CANDIDATE]: Term 4 election: Requested vote from peers 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231), e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565)
I20250905 02:22:08.747939 3205 raft_consensus.cc:2802] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 4 FOLLOWER]: Leader election won for term 4
I20250905 02:22:08.749050 3205 raft_consensus.cc:695] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 4 LEADER]: Becoming Leader. State: Replica: 27bbb6a5eec54a85b6179da4b8da31fc, State: Running, Role: LEADER
I20250905 02:22:08.751720 2986 raft_consensus.cc:2466] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27bbb6a5eec54a85b6179da4b8da31fc in term 4.
I20250905 02:22:08.750604 3205 consensus_queue.cc:237] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } }
I20250905 02:22:08.760810 3136 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:08.761333 3136 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:08.763365 3136 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077)
I20250905 02:22:08.765615 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d" is_pre_election: true
I20250905 02:22:08.766278 2838 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 2.
I20250905 02:22:08.767748 2953 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:08.766288 3104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2cb93e1290274d558112fb1c48347cba" is_pre_election: true
I20250905 02:22:08.768719 3136 raft_consensus.cc:2802] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:22:08.769140 3136 raft_consensus.cc:491] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:08.769191 3104 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 2.
I20250905 02:22:08.769506 3136 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:08.772902 2736 catalog_manager.cc:5582] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:08.777287 3136 raft_consensus.cc:513] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:08.780213 2838 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e14af3fd13ac47649ce20cae3651491d"
I20250905 02:22:08.780843 2838 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:08.788937 3136 leader_election.cc:290] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 3 election: Requested vote from peers e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565), 2cb93e1290274d558112fb1c48347cba (127.31.48.132:38077)
I20250905 02:22:08.789476 2838 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 3.
I20250905 02:22:08.790179 3104 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36694c39cdba46d9bdd426065c9426d4" candidate_uuid: "7d12f406bc4148ba854b41a9cccc2197" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2cb93e1290274d558112fb1c48347cba"
I20250905 02:22:08.790832 3104 raft_consensus.cc:3058] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:08.791014 2953 leader_election.cc:304] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d12f406bc4148ba854b41a9cccc2197, e14af3fd13ac47649ce20cae3651491d; no voters:
I20250905 02:22:08.792155 3136 raft_consensus.cc:2802] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:08.792927 3136 raft_consensus.cc:695] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 LEADER]: Becoming Leader. State: Replica: 7d12f406bc4148ba854b41a9cccc2197, State: Running, Role: LEADER
I20250905 02:22:08.793751 3136 consensus_queue.cc:237] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:08.799399 3104 raft_consensus.cc:2466] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d12f406bc4148ba854b41a9cccc2197 in term 3.
I20250905 02:22:08.812117 2735 catalog_manager.cc:5582] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 reported cstate change: term changed from 1 to 3, leader changed from e14af3fd13ac47649ce20cae3651491d (127.31.48.129) to 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131). New cstate: current_term: 3 leader_uuid: "7d12f406bc4148ba854b41a9cccc2197" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:09.043162 2838 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 02:22:09.044615 3136 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:22:09.073290 2986 raft_consensus.cc:1273] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 4 FOLLOWER]: Refusing update from remote peer 27bbb6a5eec54a85b6179da4b8da31fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 02:22:09.074754 3205 consensus_queue.cc:1035] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:09.079658 3104 raft_consensus.cc:1273] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 3 FOLLOWER]: Refusing update from remote peer 7d12f406bc4148ba854b41a9cccc2197: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250905 02:22:09.082427 3217 consensus_queue.cc:1035] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20250905 02:22:09.086609 2838 raft_consensus.cc:1273] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 4 FOLLOWER]: Refusing update from remote peer 27bbb6a5eec54a85b6179da4b8da31fc: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20250905 02:22:09.093860 3219 consensus_queue.cc:1035] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14af3fd13ac47649ce20cae3651491d" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 42565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:22:10.481812 2790 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:10.494935 2986 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642"
dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
mode: GRACEFUL
new_leader_uuid: "2cb93e1290274d558112fb1c48347cba"
from {username='slave'} at 127.0.0.1:46406
I20250905 02:22:10.495442 2986 raft_consensus.cc:604] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Received request to transfer leadership to TS 2cb93e1290274d558112fb1c48347cba
I20250905 02:22:10.496575 2790 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20250905 02:22:10.496963 2790 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:10.793484 3228 raft_consensus.cc:991] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: : Instructing follower 2cb93e1290274d558112fb1c48347cba to start an election
I20250905 02:22:10.793998 3207 raft_consensus.cc:1079] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Signalling peer 2cb93e1290274d558112fb1c48347cba to start an election
I20250905 02:22:10.795763 3104 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642"
dest_uuid: "2cb93e1290274d558112fb1c48347cba"
from {username='slave'} at 127.0.0.1:38710
I20250905 02:22:10.796411 3104 raft_consensus.cc:491] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 02:22:10.796803 3104 raft_consensus.cc:3058] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:10.804457 3104 raft_consensus.cc:513] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:10.806658 3104 leader_election.cc:290] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 election: Requested vote from peers 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981), 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131:34231)
I20250905 02:22:10.807710 2986 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "7d12f406bc4148ba854b41a9cccc2197"
I20250905 02:22:10.807610 2914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a26e4c64be9d4ecb8536269990b65642" candidate_uuid: "2cb93e1290274d558112fb1c48347cba" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "27bbb6a5eec54a85b6179da4b8da31fc"
I20250905 02:22:10.808434 2914 raft_consensus.cc:3058] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:10.808434 2986 raft_consensus.cc:3053] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 02:22:10.809067 2986 raft_consensus.cc:738] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7d12f406bc4148ba854b41a9cccc2197, State: Running, Role: LEADER
I20250905 02:22:10.810024 2986 consensus_queue.cc:260] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:10.811573 2986 raft_consensus.cc:3058] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:10.814024 2914 raft_consensus.cc:2466] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb93e1290274d558112fb1c48347cba in term 2.
I20250905 02:22:10.815068 3068 leader_election.cc:304] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27bbb6a5eec54a85b6179da4b8da31fc, 2cb93e1290274d558112fb1c48347cba; no voters:
I20250905 02:22:10.816551 3229 raft_consensus.cc:2802] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:10.817847 3229 raft_consensus.cc:695] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 2 LEADER]: Becoming Leader. State: Replica: 2cb93e1290274d558112fb1c48347cba, State: Running, Role: LEADER
I20250905 02:22:10.819303 2986 raft_consensus.cc:2466] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cb93e1290274d558112fb1c48347cba in term 2.
I20250905 02:22:10.819111 3229 consensus_queue.cc:237] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } }
I20250905 02:22:10.826421 2737 catalog_manager.cc:5582] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba reported cstate change: term changed from 1 to 2, leader changed from 7d12f406bc4148ba854b41a9cccc2197 (127.31.48.131) to 2cb93e1290274d558112fb1c48347cba (127.31.48.132). New cstate: current_term: 2 leader_uuid: "2cb93e1290274d558112fb1c48347cba" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb93e1290274d558112fb1c48347cba" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 38077 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 02:22:11.058566 2914 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Refusing update from remote peer 2cb93e1290274d558112fb1c48347cba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 02:22:11.059669 3229 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "27bbb6a5eec54a85b6179da4b8da31fc" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 32981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:22:11.074059 2986 raft_consensus.cc:1273] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Refusing update from remote peer 2cb93e1290274d558112fb1c48347cba: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 02:22:11.075309 3229 consensus_queue.cc:1035] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d12f406bc4148ba854b41a9cccc2197" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 34231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:22:12.497962 2790 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:12.500663 2790 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:12.500994 2790 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:13.077293 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:22:13.102883 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:13.103585 31938 tablet_replica.cc:331] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d: stopping tablet replica
I20250905 02:22:13.104197 31938 raft_consensus.cc:2241] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:13.104835 31938 raft_consensus.cc:2270] T 9330b5a26c9b4b3f920e33058c496be0 P e14af3fd13ac47649ce20cae3651491d [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.108007 31938 tablet_replica.cc:331] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d: stopping tablet replica
I20250905 02:22:13.108654 31938 raft_consensus.cc:2241] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:13.109359 31938 raft_consensus.cc:2270] T 36694c39cdba46d9bdd426065c9426d4 P e14af3fd13ac47649ce20cae3651491d [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.136910 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:22:13.154232 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
W20250905 02:22:13.155459 2879 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:42565: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20250905 02:22:13.160921 2879 consensus_peers.cc:489] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc -> Peer e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565): Couldn't send request to peer e14af3fd13ac47649ce20cae3651491d. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:42565: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:13.179347 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:13.180013 31938 tablet_replica.cc:331] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc: stopping tablet replica
I20250905 02:22:13.180637 31938 raft_consensus.cc:2241] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:13.181185 31938 raft_consensus.cc:2270] T a26e4c64be9d4ecb8536269990b65642 P 27bbb6a5eec54a85b6179da4b8da31fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.183279 31938 tablet_replica.cc:331] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc: stopping tablet replica
I20250905 02:22:13.183765 31938 raft_consensus.cc:2241] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 4 LEADER]: Raft consensus shutting down.
I20250905 02:22:13.184739 31938 raft_consensus.cc:2270] T 9330b5a26c9b4b3f920e33058c496be0 P 27bbb6a5eec54a85b6179da4b8da31fc [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.225024 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:22:13.239212 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:22:13.265084 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:13.265822 31938 tablet_replica.cc:331] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197: stopping tablet replica
W20250905 02:22:13.266754 2953 consensus_peers.cc:489] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 -> Peer e14af3fd13ac47649ce20cae3651491d (127.31.48.129:42565): Couldn't send request to peer e14af3fd13ac47649ce20cae3651491d. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:42565: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:13.266793 31938 raft_consensus.cc:2241] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:13.267509 31938 raft_consensus.cc:2270] T a26e4c64be9d4ecb8536269990b65642 P 7d12f406bc4148ba854b41a9cccc2197 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.269618 31938 tablet_replica.cc:331] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197: stopping tablet replica
I20250905 02:22:13.270148 31938 raft_consensus.cc:2241] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 LEADER]: Raft consensus shutting down.
I20250905 02:22:13.270989 31938 raft_consensus.cc:2270] T 36694c39cdba46d9bdd426065c9426d4 P 7d12f406bc4148ba854b41a9cccc2197 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.273149 31938 tablet_replica.cc:331] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197: stopping tablet replica
I20250905 02:22:13.273651 31938 raft_consensus.cc:2241] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:13.274117 31938 raft_consensus.cc:2270] T 9330b5a26c9b4b3f920e33058c496be0 P 7d12f406bc4148ba854b41a9cccc2197 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.299152 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:22:13.312754 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:22:13.339963 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:13.340754 31938 tablet_replica.cc:331] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba: stopping tablet replica
I20250905 02:22:13.341604 31938 raft_consensus.cc:2241] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:13.342243 31938 raft_consensus.cc:2270] T 36694c39cdba46d9bdd426065c9426d4 P 2cb93e1290274d558112fb1c48347cba [term 3 FOLLOWER]: Raft consensus is shut down!
W20250905 02:22:13.343891 3068 consensus_peers.cc:489] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba -> Peer 27bbb6a5eec54a85b6179da4b8da31fc (127.31.48.130:32981): Couldn't send request to peer 27bbb6a5eec54a85b6179da4b8da31fc. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.130:32981: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:13.344902 31938 tablet_replica.cc:331] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba: stopping tablet replica
I20250905 02:22:13.345384 31938 raft_consensus.cc:2241] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:22:13.346104 31938 raft_consensus.cc:2270] T a26e4c64be9d4ecb8536269990b65642 P 2cb93e1290274d558112fb1c48347cba [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.366930 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:22:13.380407 31938 master.cc:561] Master@127.31.48.190:36341 shutting down...
I20250905 02:22:13.399938 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:13.400602 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:13.400943 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P ccce443947d64ebe8a475a4b8bc96858: stopping tablet replica
I20250905 02:22:13.441462 31938 master.cc:583] Master@127.31.48.190:36341 shutdown complete.
[ OK ] AutoRebalancerTest.AutoRebalancingUnstableCluster (13152 ms)
[ RUN ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor
I20250905 02:22:13.481304 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:43255
I20250905 02:22:13.482527 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:13.488420 3249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.488587 3250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.489354 3252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:13.490207 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:13.491034 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:13.491305 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:13.491500 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038933491478 us; error 0 us; skew 500 ppm
I20250905 02:22:13.491995 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:13.494315 31938 webserver.cc:480] Webserver started at http://127.31.48.190:36203/ using document root <none> and password file <none>
I20250905 02:22:13.494762 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:13.494944 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:13.495193 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:13.496316 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "a8832036ce924f478c9c93d0967af6f6"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.500864 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 02:22:13.504107 3257 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.504879 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:22:13.505133 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "a8832036ce924f478c9c93d0967af6f6"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.505388 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:13.521615 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:13.522830 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:13.560458 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:43255
I20250905 02:22:13.560559 3308 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:43255 every 8 connection(s)
I20250905 02:22:13.564611 3309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:13.575524 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6: Bootstrap starting.
I20250905 02:22:13.580264 3309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:13.584625 3309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6: No bootstrap required, opened a new log
I20250905 02:22:13.586661 3309 raft_consensus.cc:357] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8832036ce924f478c9c93d0967af6f6" member_type: VOTER }
I20250905 02:22:13.587080 3309 raft_consensus.cc:383] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:13.587258 3309 raft_consensus.cc:738] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8832036ce924f478c9c93d0967af6f6, State: Initialized, Role: FOLLOWER
I20250905 02:22:13.587741 3309 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8832036ce924f478c9c93d0967af6f6" member_type: VOTER }
I20250905 02:22:13.588172 3309 raft_consensus.cc:397] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:13.588395 3309 raft_consensus.cc:491] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:13.588653 3309 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:13.592928 3309 raft_consensus.cc:513] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8832036ce924f478c9c93d0967af6f6" member_type: VOTER }
I20250905 02:22:13.593385 3309 leader_election.cc:304] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8832036ce924f478c9c93d0967af6f6; no voters:
I20250905 02:22:13.594388 3309 leader_election.cc:290] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:13.594710 3312 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:13.596027 3312 raft_consensus.cc:695] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 1 LEADER]: Becoming Leader. State: Replica: a8832036ce924f478c9c93d0967af6f6, State: Running, Role: LEADER
I20250905 02:22:13.596665 3312 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8832036ce924f478c9c93d0967af6f6" member_type: VOTER }
I20250905 02:22:13.597311 3309 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:13.601711 3313 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8832036ce924f478c9c93d0967af6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8832036ce924f478c9c93d0967af6f6" member_type: VOTER } }
I20250905 02:22:13.602794 3314 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8832036ce924f478c9c93d0967af6f6. Latest consensus state: current_term: 1 leader_uuid: "a8832036ce924f478c9c93d0967af6f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8832036ce924f478c9c93d0967af6f6" member_type: VOTER } }
I20250905 02:22:13.603435 3313 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:13.603508 3314 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:13.605636 3322 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:13.610841 3322 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:13.611814 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:22:13.619581 3322 catalog_manager.cc:1349] Generated new cluster ID: b9a19c2ed008415caa780dc16f9639f3
I20250905 02:22:13.619882 3322 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:13.636627 3322 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:13.637898 3322 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:13.653028 3322 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6: Generated new TSK 0
I20250905 02:22:13.653625 3322 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:13.678050 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:13.684015 3330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.685019 3331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.686879 3333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:13.686902 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:13.688134 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:13.688313 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:13.688503 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038933688488 us; error 0 us; skew 500 ppm
I20250905 02:22:13.688947 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:13.691304 31938 webserver.cc:480] Webserver started at http://127.31.48.129:44195/ using document root <none> and password file <none>
I20250905 02:22:13.691740 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:13.691895 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:13.692130 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:13.693295 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "0c886b356a5648d0bcc1e31b46270c82"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.698323 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:22:13.702188 3338 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.702952 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250905 02:22:13.703248 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "0c886b356a5648d0bcc1e31b46270c82"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.703493 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:13.714280 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:13.715425 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:13.716903 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:13.719201 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:13.719386 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.719635 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:13.719797 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.761003 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:35219
I20250905 02:22:13.761059 3400 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:35219 every 8 connection(s)
I20250905 02:22:13.765679 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:13.773265 3405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.775610 3406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:13.777164 31938 server_base.cc:1047] running on GCE node
W20250905 02:22:13.777210 3408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:13.778115 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:13.778321 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:13.778487 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038933778469 us; error 0 us; skew 500 ppm
I20250905 02:22:13.778971 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:13.781673 31938 webserver.cc:480] Webserver started at http://127.31.48.130:36333/ using document root <none> and password file <none>
I20250905 02:22:13.782202 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:13.782411 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:13.782712 3401 heartbeater.cc:344] Connected to a master server at 127.31.48.190:43255
I20250905 02:22:13.782779 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:13.783141 3401 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:13.783819 3401 heartbeater.cc:507] Master 127.31.48.190:43255 requested a full tablet report, sending...
I20250905 02:22:13.784003 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "35ca88be436c4d498a97493b12fbae6c"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.785894 3274 ts_manager.cc:194] Registered new tserver with Master: 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219)
I20250905 02:22:13.787678 3274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57810
I20250905 02:22:13.789055 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:13.792294 3413 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.793076 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:22:13.793345 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "35ca88be436c4d498a97493b12fbae6c"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.793596 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:13.810626 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:13.811856 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:13.813350 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:13.815526 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:13.815723 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.815954 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:13.816115 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.855940 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:44771
I20250905 02:22:13.856032 3475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:44771 every 8 connection(s)
I20250905 02:22:13.860849 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:13.867111 3479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.869055 3480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:13.872144 3482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:13.872303 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:13.873502 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:13.873797 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:13.874038 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038933874008 us; error 0 us; skew 500 ppm
I20250905 02:22:13.874624 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:13.875109 3476 heartbeater.cc:344] Connected to a master server at 127.31.48.190:43255
I20250905 02:22:13.875550 3476 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:13.876238 3476 heartbeater.cc:507] Master 127.31.48.190:43255 requested a full tablet report, sending...
I20250905 02:22:13.877506 31938 webserver.cc:480] Webserver started at http://127.31.48.131:38609/ using document root <none> and password file <none>
I20250905 02:22:13.878077 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:13.878311 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:13.878453 3274 ts_manager.cc:194] Registered new tserver with Master: 35ca88be436c4d498a97493b12fbae6c (127.31.48.130:44771)
I20250905 02:22:13.878646 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:13.880318 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "095e6f114d5442468d77d8da288eacff"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.880432 3274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57824
I20250905 02:22:13.885468 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20250905 02:22:13.888921 3487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.889668 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20250905 02:22:13.889925 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "095e6f114d5442468d77d8da288eacff"
format_stamp: "Formatted at 2025-09-05 02:22:13 on dist-test-slave-q32q"
I20250905 02:22:13.890177 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:13.913798 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:13.914927 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:13.916306 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:13.918707 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:13.919013 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.919310 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:13.919487 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:13.958122 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:45149
I20250905 02:22:13.958215 3549 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:45149 every 8 connection(s)
I20250905 02:22:13.971093 3550 heartbeater.cc:344] Connected to a master server at 127.31.48.190:43255
I20250905 02:22:13.971446 3550 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:13.972144 3550 heartbeater.cc:507] Master 127.31.48.190:43255 requested a full tablet report, sending...
I20250905 02:22:13.973915 3274 ts_manager.cc:194] Registered new tserver with Master: 095e6f114d5442468d77d8da288eacff (127.31.48.131:45149)
I20250905 02:22:13.974583 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013211538s
I20250905 02:22:13.975257 3274 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:57836
I20250905 02:22:14.790305 3401 heartbeater.cc:499] Master 127.31.48.190:43255 was elected leader, sending a full tablet report...
I20250905 02:22:14.883136 3476 heartbeater.cc:499] Master 127.31.48.190:43255 was elected leader, sending a full tablet report...
I20250905 02:22:14.977890 3550 heartbeater.cc:499] Master 127.31.48.190:43255 was elected leader, sending a full tablet report...
I20250905 02:22:15.007031 31938 test_util.cc:276] Using random seed: -2121015521
I20250905 02:22:15.028570 3274 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57852:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:22:15.030594 3274 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:22:15.079283 3515 tablet_service.cc:1468] Processing CreateTablet for tablet c5970638b3dd40eb91f6260013fc32c7 (DEFAULT_TABLE table=test-workload [id=e25982f3fe124df682364da33815d10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:15.080483 3515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5970638b3dd40eb91f6260013fc32c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:15.085829 3441 tablet_service.cc:1468] Processing CreateTablet for tablet c5970638b3dd40eb91f6260013fc32c7 (DEFAULT_TABLE table=test-workload [id=e25982f3fe124df682364da33815d10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:15.087185 3441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5970638b3dd40eb91f6260013fc32c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:15.092690 3366 tablet_service.cc:1468] Processing CreateTablet for tablet c5970638b3dd40eb91f6260013fc32c7 (DEFAULT_TABLE table=test-workload [id=e25982f3fe124df682364da33815d10b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 02:22:15.093978 3366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5970638b3dd40eb91f6260013fc32c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:15.097035 3570 tablet_bootstrap.cc:492] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: Bootstrap starting.
I20250905 02:22:15.102716 3570 tablet_bootstrap.cc:654] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:15.105150 3571 tablet_bootstrap.cc:492] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c: Bootstrap starting.
I20250905 02:22:15.112797 3573 tablet_bootstrap.cc:492] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82: Bootstrap starting.
I20250905 02:22:15.115011 3570 tablet_bootstrap.cc:492] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: No bootstrap required, opened a new log
I20250905 02:22:15.115501 3571 tablet_bootstrap.cc:654] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:15.115541 3570 ts_tablet_manager.cc:1397] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.007s
I20250905 02:22:15.117895 3573 tablet_bootstrap.cc:654] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:15.118834 3570 raft_consensus.cc:357] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.119880 3570 raft_consensus.cc:383] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:15.120323 3570 raft_consensus.cc:738] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095e6f114d5442468d77d8da288eacff, State: Initialized, Role: FOLLOWER
I20250905 02:22:15.121574 3570 consensus_queue.cc:260] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.123664 3571 tablet_bootstrap.cc:492] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c: No bootstrap required, opened a new log
I20250905 02:22:15.124110 3571 ts_tablet_manager.cc:1397] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.007s
I20250905 02:22:15.124177 3573 tablet_bootstrap.cc:492] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82: No bootstrap required, opened a new log
I20250905 02:22:15.124737 3573 ts_tablet_manager.cc:1397] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20250905 02:22:15.125392 3570 ts_tablet_manager.cc:1428] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20250905 02:22:15.126646 3571 raft_consensus.cc:357] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.127195 3571 raft_consensus.cc:383] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:15.127444 3571 raft_consensus.cc:738] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ca88be436c4d498a97493b12fbae6c, State: Initialized, Role: FOLLOWER
I20250905 02:22:15.127385 3573 raft_consensus.cc:357] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.128306 3573 raft_consensus.cc:383] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:15.128185 3571 consensus_queue.cc:260] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.128670 3573 raft_consensus.cc:738] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c886b356a5648d0bcc1e31b46270c82, State: Initialized, Role: FOLLOWER
I20250905 02:22:15.129474 3573 consensus_queue.cc:260] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.131197 3571 ts_tablet_manager.cc:1428] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 02:22:15.141528 3573 ts_tablet_manager.cc:1428] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82: Time spent starting tablet: real 0.016s user 0.008s sys 0.004s
W20250905 02:22:15.213474 3551 tablet.cc:2378] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:22:15.244095 3577 raft_consensus.cc:491] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:15.244613 3577 raft_consensus.cc:513] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.246729 3577 leader_election.cc:290] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219), 095e6f114d5442468d77d8da288eacff (127.31.48.131:45149)
I20250905 02:22:15.259537 3525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5970638b3dd40eb91f6260013fc32c7" candidate_uuid: "35ca88be436c4d498a97493b12fbae6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095e6f114d5442468d77d8da288eacff" is_pre_election: true
I20250905 02:22:15.259546 3376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5970638b3dd40eb91f6260013fc32c7" candidate_uuid: "35ca88be436c4d498a97493b12fbae6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c886b356a5648d0bcc1e31b46270c82" is_pre_election: true
I20250905 02:22:15.260612 3376 raft_consensus.cc:2466] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ca88be436c4d498a97493b12fbae6c in term 0.
I20250905 02:22:15.260612 3525 raft_consensus.cc:2466] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ca88be436c4d498a97493b12fbae6c in term 0.
I20250905 02:22:15.262180 3417 leader_election.cc:304] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095e6f114d5442468d77d8da288eacff, 35ca88be436c4d498a97493b12fbae6c; no voters:
I20250905 02:22:15.262928 3577 raft_consensus.cc:2802] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:15.263258 3577 raft_consensus.cc:491] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:15.263542 3577 raft_consensus.cc:3058] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:15.269342 3577 raft_consensus.cc:513] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.270859 3577 leader_election.cc:290] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [CANDIDATE]: Term 1 election: Requested vote from peers 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219), 095e6f114d5442468d77d8da288eacff (127.31.48.131:45149)
I20250905 02:22:15.271838 3376 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5970638b3dd40eb91f6260013fc32c7" candidate_uuid: "35ca88be436c4d498a97493b12fbae6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c886b356a5648d0bcc1e31b46270c82"
I20250905 02:22:15.271837 3525 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c5970638b3dd40eb91f6260013fc32c7" candidate_uuid: "35ca88be436c4d498a97493b12fbae6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "095e6f114d5442468d77d8da288eacff"
I20250905 02:22:15.272429 3376 raft_consensus.cc:3058] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:15.272545 3525 raft_consensus.cc:3058] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:15.277441 3376 raft_consensus.cc:2466] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ca88be436c4d498a97493b12fbae6c in term 1.
I20250905 02:22:15.277438 3525 raft_consensus.cc:2466] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ca88be436c4d498a97493b12fbae6c in term 1.
I20250905 02:22:15.278579 3416 leader_election.cc:304] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c886b356a5648d0bcc1e31b46270c82, 35ca88be436c4d498a97493b12fbae6c; no voters:
I20250905 02:22:15.279263 3577 raft_consensus.cc:2802] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:15.280509 3577 raft_consensus.cc:695] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 1 LEADER]: Becoming Leader. State: Replica: 35ca88be436c4d498a97493b12fbae6c, State: Running, Role: LEADER
I20250905 02:22:15.281389 3577 consensus_queue.cc:237] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } }
I20250905 02:22:15.289242 3272 catalog_manager.cc:5582] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ca88be436c4d498a97493b12fbae6c (127.31.48.130). New cstate: current_term: 1 leader_uuid: "35ca88be436c4d498a97493b12fbae6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ca88be436c4d498a97493b12fbae6c" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c886b356a5648d0bcc1e31b46270c82" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:15.331349 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:22:15.351692 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:15.352471 31938 tablet_replica.cc:331] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82: stopping tablet replica
I20250905 02:22:15.353035 31938 raft_consensus.cc:2241] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:15.353439 31938 raft_consensus.cc:2270] T c5970638b3dd40eb91f6260013fc32c7 P 0c886b356a5648d0bcc1e31b46270c82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:15.371379 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:22:15.611532 3327 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:15.612927 3327 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:15.613293 3327 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20250905 02:22:15.740710 3577 consensus_queue.cc:1035] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "095e6f114d5442468d77d8da288eacff" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 45149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:22:15.757176 3416 consensus_peers.cc:489] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c -> Peer 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219): Couldn't send request to peer 0c886b356a5648d0bcc1e31b46270c82. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:17.614199 3327 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:17.615586 3327 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:17.615914 3327 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 02:22:18.347450 3416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20250905 02:22:18.350556 3416 consensus_peers.cc:489] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c -> Peer 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219): Couldn't send request to peer 0c886b356a5648d0bcc1e31b46270c82. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250905 02:22:19.616868 3327 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:19.618075 3327 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:19.618357 3327 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 02:22:20.673739 3416 consensus_peers.cc:489] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c -> Peer 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219): Couldn't send request to peer 0c886b356a5648d0bcc1e31b46270c82. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250905 02:22:21.619581 3327 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:21.621374 3327 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:21.621639 3327 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 02:22:23.315508 3416 consensus_peers.cc:489] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c -> Peer 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219): Couldn't send request to peer 0c886b356a5648d0bcc1e31b46270c82. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250905 02:22:23.622507 3327 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:23.623677 3327 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:23.623915 3327 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 02:22:23.880213 3416 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250905 02:22:25.624760 3327 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20250905 02:22:25.625911 3327 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20250905 02:22:25.626159 3327 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
W20250905 02:22:26.002417 3416 consensus_peers.cc:489] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c -> Peer 0c886b356a5648d0bcc1e31b46270c82 (127.31.48.129:35219): Couldn't send request to peer 0c886b356a5648d0bcc1e31b46270c82. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35219: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250905 02:22:26.442278 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:22:26.463181 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:26.463843 31938 tablet_replica.cc:331] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c: stopping tablet replica
I20250905 02:22:26.464351 31938 raft_consensus.cc:2241] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:26.465348 31938 raft_consensus.cc:2270] T c5970638b3dd40eb91f6260013fc32c7 P 35ca88be436c4d498a97493b12fbae6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:26.485782 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:22:26.498508 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:22:26.518603 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:26.519219 31938 tablet_replica.cc:331] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff: stopping tablet replica
I20250905 02:22:26.519776 31938 raft_consensus.cc:2241] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:26.520246 31938 raft_consensus.cc:2270] T c5970638b3dd40eb91f6260013fc32c7 P 095e6f114d5442468d77d8da288eacff [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:26.539165 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:22:26.549779 31938 master.cc:561] Master@127.31.48.190:43255 shutting down...
I20250905 02:22:26.566798 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:26.567420 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:26.567731 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P a8832036ce924f478c9c93d0967af6f6: stopping tablet replica
I20250905 02:22:26.586017 31938 master.cc:583] Master@127.31.48.190:43255 shutdown complete.
[ OK ] AutoRebalancerTest.NoReplicaMovesIfCannotMeetReplicationFactor (13140 ms)
[ RUN ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering
I20250905 02:22:26.621793 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:36911
I20250905 02:22:26.622910 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:26.628772 3615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:26.629235 3618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:26.628827 3616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:26.631330 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:26.632097 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:26.632295 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:26.632505 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038946632479 us; error 0 us; skew 500 ppm
I20250905 02:22:26.632985 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:26.635321 31938 webserver.cc:480] Webserver started at http://127.31.48.190:41221/ using document root <none> and password file <none>
I20250905 02:22:26.635780 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:26.635962 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:26.636214 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:26.637605 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "c863edee63be4b13b7846bc562900dde"
format_stamp: "Formatted at 2025-09-05 02:22:26 on dist-test-slave-q32q"
I20250905 02:22:26.642204 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250905 02:22:26.645686 3623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:26.646464 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20250905 02:22:26.646731 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "c863edee63be4b13b7846bc562900dde"
format_stamp: "Formatted at 2025-09-05 02:22:26 on dist-test-slave-q32q"
I20250905 02:22:26.646984 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:26.670018 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:26.671262 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:26.707849 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:36911
I20250905 02:22:26.707957 3674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:36911 every 8 connection(s)
I20250905 02:22:26.711791 3675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:26.722890 3675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde: Bootstrap starting.
I20250905 02:22:26.727293 3675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:26.731449 3675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde: No bootstrap required, opened a new log
I20250905 02:22:26.733376 3675 raft_consensus.cc:357] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863edee63be4b13b7846bc562900dde" member_type: VOTER }
I20250905 02:22:26.733777 3675 raft_consensus.cc:383] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:26.733958 3675 raft_consensus.cc:738] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c863edee63be4b13b7846bc562900dde, State: Initialized, Role: FOLLOWER
I20250905 02:22:26.734488 3675 consensus_queue.cc:260] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863edee63be4b13b7846bc562900dde" member_type: VOTER }
I20250905 02:22:26.734926 3675 raft_consensus.cc:397] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:26.735117 3675 raft_consensus.cc:491] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:26.735342 3675 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:26.739655 3675 raft_consensus.cc:513] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863edee63be4b13b7846bc562900dde" member_type: VOTER }
I20250905 02:22:26.740113 3675 leader_election.cc:304] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c863edee63be4b13b7846bc562900dde; no voters:
I20250905 02:22:26.741206 3675 leader_election.cc:290] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:26.741506 3678 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:26.742726 3678 raft_consensus.cc:695] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 1 LEADER]: Becoming Leader. State: Replica: c863edee63be4b13b7846bc562900dde, State: Running, Role: LEADER
I20250905 02:22:26.743391 3678 consensus_queue.cc:237] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863edee63be4b13b7846bc562900dde" member_type: VOTER }
I20250905 02:22:26.744092 3675 sys_catalog.cc:564] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:26.746353 3679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c863edee63be4b13b7846bc562900dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863edee63be4b13b7846bc562900dde" member_type: VOTER } }
I20250905 02:22:26.746462 3680 sys_catalog.cc:455] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [sys.catalog]: SysCatalogTable state changed. Reason: New leader c863edee63be4b13b7846bc562900dde. Latest consensus state: current_term: 1 leader_uuid: "c863edee63be4b13b7846bc562900dde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c863edee63be4b13b7846bc562900dde" member_type: VOTER } }
I20250905 02:22:26.747104 3680 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:26.747046 3679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:26.754163 3684 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:26.759399 3684 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:26.760387 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:22:26.768519 3684 catalog_manager.cc:1349] Generated new cluster ID: a6de6650472b4a4bb79bcc6611826e7f
I20250905 02:22:26.768812 3684 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:26.794369 3684 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:26.795675 3684 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:26.810851 3684 catalog_manager.cc:5955] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde: Generated new TSK 0
I20250905 02:22:26.811539 3684 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:26.826725 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:26.834900 3697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:26.834820 3696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:26.840312 3699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:26.840350 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:26.841748 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:26.842020 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:26.842191 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038946842172 us; error 0 us; skew 500 ppm
I20250905 02:22:26.842808 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:26.845237 31938 webserver.cc:480] Webserver started at http://127.31.48.129:44713/ using document root <none> and password file <none>
I20250905 02:22:26.845693 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:26.845885 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:26.846131 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:26.847191 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "27a038d5acd0402a970fcd96c8ed01be"
format_stamp: "Formatted at 2025-09-05 02:22:26 on dist-test-slave-q32q"
I20250905 02:22:26.851656 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:22:26.854885 3704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:26.855621 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:22:26.855878 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "27a038d5acd0402a970fcd96c8ed01be"
format_stamp: "Formatted at 2025-09-05 02:22:26 on dist-test-slave-q32q"
I20250905 02:22:26.856139 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:26.876763 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:26.877907 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:26.879359 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:26.881686 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:26.881878 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:26.882115 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:26.882278 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:26.921729 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:36291
I20250905 02:22:26.921830 3766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:36291 every 8 connection(s)
I20250905 02:22:26.927031 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:26.933827 3771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:26.934278 3772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:26.937131 31938 server_base.cc:1047] running on GCE node
W20250905 02:22:26.939458 3774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:26.940459 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 02:22:26.940481 3767 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36911
W20250905 02:22:26.940877 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:26.941022 3767 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:26.941119 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038946941098 us; error 0 us; skew 500 ppm
I20250905 02:22:26.941916 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:26.941946 3767 heartbeater.cc:507] Master 127.31.48.190:36911 requested a full tablet report, sending...
I20250905 02:22:26.944218 3640 ts_manager.cc:194] Registered new tserver with Master: 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291)
I20250905 02:22:26.944797 31938 webserver.cc:480] Webserver started at http://127.31.48.130:37055/ using document root <none> and password file <none>
I20250905 02:22:26.945312 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:26.945530 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:26.945820 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:26.946239 3640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52134
I20250905 02:22:26.947018 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "381c7bc182e94ca9a069646bac9236fd"
format_stamp: "Formatted at 2025-09-05 02:22:26 on dist-test-slave-q32q"
I20250905 02:22:26.951920 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20250905 02:22:26.955220 3779 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:26.956044 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 02:22:26.956399 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "381c7bc182e94ca9a069646bac9236fd"
format_stamp: "Formatted at 2025-09-05 02:22:26 on dist-test-slave-q32q"
I20250905 02:22:26.956670 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:26.977600 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:26.978754 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:26.980197 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:26.982333 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:26.982529 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:26.982743 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:26.982890 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:27.020939 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:39449
I20250905 02:22:27.021031 3841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:39449 every 8 connection(s)
I20250905 02:22:27.025578 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:27.032490 3845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:27.034417 3846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:27.036516 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:27.036725 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:27.037168 3842 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36911
I20250905 02:22:27.037557 3842 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:27.037830 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:27.038059 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:27.038236 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038947038219 us; error 0 us; skew 500 ppm
I20250905 02:22:27.038277 3842 heartbeater.cc:507] Master 127.31.48.190:36911 requested a full tablet report, sending...
I20250905 02:22:27.038766 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:27.040449 3640 ts_manager.cc:194] Registered new tserver with Master: 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:27.041491 31938 webserver.cc:480] Webserver started at http://127.31.48.131:36019/ using document root <none> and password file <none>
I20250905 02:22:27.042106 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:27.042354 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:27.042290 3640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52140
I20250905 02:22:27.042819 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:27.044318 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "a384e9e80f10425c850acd9d8d61b45f"
format_stamp: "Formatted at 2025-09-05 02:22:27 on dist-test-slave-q32q"
I20250905 02:22:27.049001 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:27.052098 3853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:27.052997 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:22:27.053305 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "a384e9e80f10425c850acd9d8d61b45f"
format_stamp: "Formatted at 2025-09-05 02:22:27 on dist-test-slave-q32q"
I20250905 02:22:27.053566 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:27.070681 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:27.071862 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:27.073344 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:27.075466 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:27.075672 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:27.075879 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:27.076028 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:27.114755 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:39369
I20250905 02:22:27.114848 3915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:39369 every 8 connection(s)
I20250905 02:22:27.128005 3916 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36911
I20250905 02:22:27.128434 3916 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:27.129104 3916 heartbeater.cc:507] Master 127.31.48.190:36911 requested a full tablet report, sending...
I20250905 02:22:27.130873 3640 ts_manager.cc:194] Registered new tserver with Master: a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:27.131959 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014121621s
I20250905 02:22:27.132229 3640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52142
I20250905 02:22:27.949133 3767 heartbeater.cc:499] Master 127.31.48.190:36911 was elected leader, sending a full tablet report...
I20250905 02:22:28.045352 3842 heartbeater.cc:499] Master 127.31.48.190:36911 was elected leader, sending a full tablet report...
I20250905 02:22:28.134899 3916 heartbeater.cc:499] Master 127.31.48.190:36911 was elected leader, sending a full tablet report...
I20250905 02:22:28.164559 31938 test_util.cc:276] Using random seed: -2107857993
I20250905 02:22:28.187774 3640 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52150:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:22:28.190039 3640 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:22:28.258977 3881 tablet_service.cc:1468] Processing CreateTablet for tablet 556a3d6954084dc481464d12488767f1 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:22:28.260152 3880 tablet_service.cc:1468] Processing CreateTablet for tablet 485c44a581a941ad9d73b6b68057a359 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:22:28.261590 3804 tablet_service.cc:1468] Processing CreateTablet for tablet 78fde155a8eb49e9898595529eb6cce0 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:22:28.262346 3881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556a3d6954084dc481464d12488767f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.262382 3879 tablet_service.cc:1468] Processing CreateTablet for tablet be7336e3b2e14e5ca2225619988bd6d6 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:22:28.263530 3878 tablet_service.cc:1468] Processing CreateTablet for tablet 78fde155a8eb49e9898595529eb6cce0 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:22:28.263912 3879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7336e3b2e14e5ca2225619988bd6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.261523 3807 tablet_service.cc:1468] Processing CreateTablet for tablet 556a3d6954084dc481464d12488767f1 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:22:28.262971 3804 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78fde155a8eb49e9898595529eb6cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.264806 3878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78fde155a8eb49e9898595529eb6cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.265588 3807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556a3d6954084dc481464d12488767f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.266265 3806 tablet_service.cc:1468] Processing CreateTablet for tablet 485c44a581a941ad9d73b6b68057a359 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:22:28.267482 3806 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c44a581a941ad9d73b6b68057a359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.264460 3805 tablet_service.cc:1468] Processing CreateTablet for tablet be7336e3b2e14e5ca2225619988bd6d6 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:22:28.273375 3805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7336e3b2e14e5ca2225619988bd6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.280786 3880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c44a581a941ad9d73b6b68057a359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.296133 3732 tablet_service.cc:1468] Processing CreateTablet for tablet 556a3d6954084dc481464d12488767f1 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:22:28.297667 3732 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556a3d6954084dc481464d12488767f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.301734 3731 tablet_service.cc:1468] Processing CreateTablet for tablet 485c44a581a941ad9d73b6b68057a359 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:22:28.303244 3731 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c44a581a941ad9d73b6b68057a359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.305786 3730 tablet_service.cc:1468] Processing CreateTablet for tablet be7336e3b2e14e5ca2225619988bd6d6 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:22:28.306854 3936 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd: Bootstrap starting.
I20250905 02:22:28.310088 3729 tablet_service.cc:1468] Processing CreateTablet for tablet 78fde155a8eb49e9898595529eb6cce0 (DEFAULT_TABLE table=test-workload [id=645d4373ef35443ea2557f2125fa1b67]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:22:28.311475 3730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7336e3b2e14e5ca2225619988bd6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.316717 3729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78fde155a8eb49e9898595529eb6cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:28.316690 3936 tablet_bootstrap.cc:654] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.347750 3937 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f: Bootstrap starting.
I20250905 02:22:28.351784 3939 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be: Bootstrap starting.
I20250905 02:22:28.350596 3936 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd: No bootstrap required, opened a new log
I20250905 02:22:28.352553 3936 ts_tablet_manager.cc:1397] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.013s
I20250905 02:22:28.355405 3936 raft_consensus.cc:357] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.356266 3936 raft_consensus.cc:383] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.356693 3936 raft_consensus.cc:738] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381c7bc182e94ca9a069646bac9236fd, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.357625 3936 consensus_queue.cc:260] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.359551 3939 tablet_bootstrap.cc:654] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.360146 3937 tablet_bootstrap.cc:654] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.366043 3936 ts_tablet_manager.cc:1428] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20250905 02:22:28.367161 3936 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: Bootstrap starting.
I20250905 02:22:28.367950 3939 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be: No bootstrap required, opened a new log
I20250905 02:22:28.368345 3937 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f: No bootstrap required, opened a new log
I20250905 02:22:28.368542 3939 ts_tablet_manager.cc:1397] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20250905 02:22:28.368839 3937 ts_tablet_manager.cc:1397] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.009s
I20250905 02:22:28.371567 3939 raft_consensus.cc:357] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.371820 3937 raft_consensus.cc:357] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.372555 3937 raft_consensus.cc:383] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.372514 3939 raft_consensus.cc:383] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.372990 3937 raft_consensus.cc:738] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a384e9e80f10425c850acd9d8d61b45f, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.373099 3939 raft_consensus.cc:738] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a038d5acd0402a970fcd96c8ed01be, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.373387 3936 tablet_bootstrap.cc:654] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.373802 3937 consensus_queue.cc:260] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.373987 3939 consensus_queue.cc:260] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.380329 3939 ts_tablet_manager.cc:1428] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20250905 02:22:28.381705 3939 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: Bootstrap starting.
I20250905 02:22:28.381760 3936 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: No bootstrap required, opened a new log
I20250905 02:22:28.382439 3936 ts_tablet_manager.cc:1397] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.006s
I20250905 02:22:28.385586 3937 ts_tablet_manager.cc:1428] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f: Time spent starting tablet: real 0.016s user 0.009s sys 0.003s
I20250905 02:22:28.386718 3937 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: Bootstrap starting.
I20250905 02:22:28.388263 3939 tablet_bootstrap.cc:654] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.387969 3936 raft_consensus.cc:357] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.388762 3936 raft_consensus.cc:383] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.389072 3936 raft_consensus.cc:738] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381c7bc182e94ca9a069646bac9236fd, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.389925 3936 consensus_queue.cc:260] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.393911 3936 ts_tablet_manager.cc:1428] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: Time spent starting tablet: real 0.011s user 0.004s sys 0.002s
I20250905 02:22:28.393966 3937 tablet_bootstrap.cc:654] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.394996 3936 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd: Bootstrap starting.
I20250905 02:22:28.401620 3936 tablet_bootstrap.cc:654] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.407553 3939 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: No bootstrap required, opened a new log
I20250905 02:22:28.408025 3939 ts_tablet_manager.cc:1397] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.015s
I20250905 02:22:28.410027 3939 raft_consensus.cc:357] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.410580 3939 raft_consensus.cc:383] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.410894 3939 raft_consensus.cc:738] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a038d5acd0402a970fcd96c8ed01be, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.411757 3939 consensus_queue.cc:260] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.413761 3936 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd: No bootstrap required, opened a new log
I20250905 02:22:28.414095 3939 ts_tablet_manager.cc:1428] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20250905 02:22:28.414413 3936 ts_tablet_manager.cc:1397] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.003s
I20250905 02:22:28.415067 3939 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be: Bootstrap starting.
I20250905 02:22:28.417330 3936 raft_consensus.cc:357] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.418129 3936 raft_consensus.cc:383] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.418500 3936 raft_consensus.cc:738] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381c7bc182e94ca9a069646bac9236fd, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.419287 3936 consensus_queue.cc:260] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.421695 3939 tablet_bootstrap.cc:654] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.423980 3936 ts_tablet_manager.cc:1428] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250905 02:22:28.425361 3936 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: Bootstrap starting.
I20250905 02:22:28.434835 3936 tablet_bootstrap.cc:654] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.435446 3939 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be: No bootstrap required, opened a new log
I20250905 02:22:28.436188 3939 ts_tablet_manager.cc:1397] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.013s
I20250905 02:22:28.440047 3939 raft_consensus.cc:357] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.441157 3939 raft_consensus.cc:383] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.441546 3939 raft_consensus.cc:738] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a038d5acd0402a970fcd96c8ed01be, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.442684 3939 consensus_queue.cc:260] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.448719 3939 ts_tablet_manager.cc:1428] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20250905 02:22:28.450335 3939 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: Bootstrap starting.
I20250905 02:22:28.459501 3939 tablet_bootstrap.cc:654] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.470513 3936 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: No bootstrap required, opened a new log
I20250905 02:22:28.470434 3939 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: No bootstrap required, opened a new log
I20250905 02:22:28.470434 3937 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: No bootstrap required, opened a new log
I20250905 02:22:28.471238 3939 ts_tablet_manager.cc:1397] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.009s
I20250905 02:22:28.471355 3937 ts_tablet_manager.cc:1397] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: Time spent bootstrapping tablet: real 0.085s user 0.025s sys 0.054s
I20250905 02:22:28.471248 3936 ts_tablet_manager.cc:1397] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.023s
I20250905 02:22:28.473594 3939 raft_consensus.cc:357] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.474131 3939 raft_consensus.cc:383] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.474413 3939 raft_consensus.cc:738] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27a038d5acd0402a970fcd96c8ed01be, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.474452 3937 raft_consensus.cc:357] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.474682 3936 raft_consensus.cc:357] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.475260 3937 raft_consensus.cc:383] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.475342 3936 raft_consensus.cc:383] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.475641 3937 raft_consensus.cc:738] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a384e9e80f10425c850acd9d8d61b45f, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.475404 3939 consensus_queue.cc:260] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.475831 3936 raft_consensus.cc:738] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 381c7bc182e94ca9a069646bac9236fd, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.476434 3937 consensus_queue.cc:260] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.477497 3939 ts_tablet_manager.cc:1428] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250905 02:22:28.478621 3937 ts_tablet_manager.cc:1428] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250905 02:22:28.478634 3936 consensus_queue.cc:260] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.479692 3937 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f: Bootstrap starting.
I20250905 02:22:28.481069 3936 ts_tablet_manager.cc:1428] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250905 02:22:28.485910 3937 tablet_bootstrap.cc:654] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.490430 3937 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f: No bootstrap required, opened a new log
I20250905 02:22:28.490882 3937 ts_tablet_manager.cc:1397] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20250905 02:22:28.493126 3937 raft_consensus.cc:357] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.493624 3937 raft_consensus.cc:383] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.493898 3937 raft_consensus.cc:738] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a384e9e80f10425c850acd9d8d61b45f, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.494443 3937 consensus_queue.cc:260] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.496354 3937 ts_tablet_manager.cc:1428] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 02:22:28.497321 3937 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: Bootstrap starting.
I20250905 02:22:28.501749 3937 tablet_bootstrap.cc:654] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:28.501963 3944 raft_consensus.cc:491] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:28.502455 3944 raft_consensus.cc:513] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.506345 3944 leader_election.cc:290] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449), a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:28.507391 3940 raft_consensus.cc:491] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:28.507870 3940 raft_consensus.cc:513] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.519393 3940 leader_election.cc:290] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369), 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291)
I20250905 02:22:28.525285 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78fde155a8eb49e9898595529eb6cce0" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd" is_pre_election: true
I20250905 02:22:28.526185 3817 raft_consensus.cc:2466] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 0.
I20250905 02:22:28.529603 3707 leader_election.cc:304] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, 381c7bc182e94ca9a069646bac9236fd; no voters:
I20250905 02:22:28.530547 3944 raft_consensus.cc:2802] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:28.530936 3944 raft_consensus.cc:491] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:28.531240 3944 raft_consensus.cc:3058] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.531246 3937 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: No bootstrap required, opened a new log
I20250905 02:22:28.531996 3937 ts_tablet_manager.cc:1397] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.025s
I20250905 02:22:28.533393 3891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78fde155a8eb49e9898595529eb6cce0" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f" is_pre_election: true
I20250905 02:22:28.534078 3891 raft_consensus.cc:2466] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 0.
I20250905 02:22:28.534890 3937 raft_consensus.cc:357] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.535568 3937 raft_consensus.cc:383] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:28.535848 3937 raft_consensus.cc:738] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a384e9e80f10425c850acd9d8d61b45f, State: Initialized, Role: FOLLOWER
I20250905 02:22:28.536473 3937 consensus_queue.cc:260] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.538841 3937 ts_tablet_manager.cc:1428] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20250905 02:22:28.541630 3742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "556a3d6954084dc481464d12488767f1" candidate_uuid: "381c7bc182e94ca9a069646bac9236fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a038d5acd0402a970fcd96c8ed01be" is_pre_election: true
I20250905 02:22:28.542234 3891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "556a3d6954084dc481464d12488767f1" candidate_uuid: "381c7bc182e94ca9a069646bac9236fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f" is_pre_election: true
I20250905 02:22:28.542325 3742 raft_consensus.cc:2466] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 381c7bc182e94ca9a069646bac9236fd in term 0.
I20250905 02:22:28.542804 3891 raft_consensus.cc:2466] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 381c7bc182e94ca9a069646bac9236fd in term 0.
I20250905 02:22:28.543706 3780 leader_election.cc:304] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, 381c7bc182e94ca9a069646bac9236fd; no voters:
I20250905 02:22:28.544386 3944 raft_consensus.cc:513] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.544664 3940 raft_consensus.cc:2802] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:28.545183 3940 raft_consensus.cc:491] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:28.545562 3940 raft_consensus.cc:3058] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.546355 3944 leader_election.cc:290] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 election: Requested vote from peers 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449), a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:28.546891 3944 raft_consensus.cc:491] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:28.547151 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78fde155a8eb49e9898595529eb6cce0" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd"
I20250905 02:22:28.547410 3891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78fde155a8eb49e9898595529eb6cce0" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f"
I20250905 02:22:28.548056 3891 raft_consensus.cc:3058] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.548045 3817 raft_consensus.cc:3058] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.555651 3817 raft_consensus.cc:2466] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 1.
I20250905 02:22:28.555670 3891 raft_consensus.cc:2466] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 1.
I20250905 02:22:28.555883 3940 raft_consensus.cc:513] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.556885 3707 leader_election.cc:304] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, 381c7bc182e94ca9a069646bac9236fd; no voters:
I20250905 02:22:28.555752 3944 raft_consensus.cc:513] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.558786 3944 leader_election.cc:290] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449), a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:28.559288 3944 raft_consensus.cc:2802] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:28.559705 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7336e3b2e14e5ca2225619988bd6d6" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd" is_pre_election: true
I20250905 02:22:28.560128 3891 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7336e3b2e14e5ca2225619988bd6d6" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f" is_pre_election: true
I20250905 02:22:28.560508 3817 raft_consensus.cc:2466] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 0.
I20250905 02:22:28.560845 3891 raft_consensus.cc:2466] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 0.
I20250905 02:22:28.561699 3707 leader_election.cc:304] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, 381c7bc182e94ca9a069646bac9236fd; no voters:
I20250905 02:22:28.561966 3940 leader_election.cc:290] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 1 election: Requested vote from peers a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369), 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291)
I20250905 02:22:28.561980 3890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "556a3d6954084dc481464d12488767f1" candidate_uuid: "381c7bc182e94ca9a069646bac9236fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f"
I20250905 02:22:28.559710 3944 raft_consensus.cc:695] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 1 LEADER]: Becoming Leader. State: Replica: 27a038d5acd0402a970fcd96c8ed01be, State: Running, Role: LEADER
I20250905 02:22:28.563230 3742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "556a3d6954084dc481464d12488767f1" candidate_uuid: "381c7bc182e94ca9a069646bac9236fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a038d5acd0402a970fcd96c8ed01be"
I20250905 02:22:28.563730 3742 raft_consensus.cc:3058] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.562812 3890 raft_consensus.cc:3058] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.564086 3944 consensus_queue.cc:237] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.571197 3742 raft_consensus.cc:2466] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381c7bc182e94ca9a069646bac9236fd in term 1.
I20250905 02:22:28.562805 3959 raft_consensus.cc:2802] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:28.572320 3959 raft_consensus.cc:491] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:28.572520 3780 leader_election.cc:304] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, 381c7bc182e94ca9a069646bac9236fd; no voters:
I20250905 02:22:28.572875 3959 raft_consensus.cc:3058] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.571686 3890 raft_consensus.cc:2466] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381c7bc182e94ca9a069646bac9236fd in term 1.
I20250905 02:22:28.574983 3940 raft_consensus.cc:2802] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:28.582623 3959 raft_consensus.cc:513] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.585271 3639 catalog_manager.cc:5582] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be reported cstate change: term changed from 0 to 1, leader changed from <none> to 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129). New cstate: current_term: 1 leader_uuid: "27a038d5acd0402a970fcd96c8ed01be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:28.586320 3959 leader_election.cc:290] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 election: Requested vote from peers 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449), a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:28.587051 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7336e3b2e14e5ca2225619988bd6d6" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd"
I20250905 02:22:28.587324 3890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7336e3b2e14e5ca2225619988bd6d6" candidate_uuid: "27a038d5acd0402a970fcd96c8ed01be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f"
I20250905 02:22:28.587659 3817 raft_consensus.cc:3058] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.587931 3890 raft_consensus.cc:3058] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.590058 3940 raft_consensus.cc:695] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 LEADER]: Becoming Leader. State: Replica: 381c7bc182e94ca9a069646bac9236fd, State: Running, Role: LEADER
I20250905 02:22:28.591094 3940 consensus_queue.cc:237] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } }
I20250905 02:22:28.594684 3817 raft_consensus.cc:2466] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 1.
I20250905 02:22:28.594815 3890 raft_consensus.cc:2466] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27a038d5acd0402a970fcd96c8ed01be in term 1.
I20250905 02:22:28.595755 3707 leader_election.cc:304] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, 381c7bc182e94ca9a069646bac9236fd; no voters:
I20250905 02:22:28.596836 3944 raft_consensus.cc:2802] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:28.597352 3944 raft_consensus.cc:695] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 1 LEADER]: Becoming Leader. State: Replica: 27a038d5acd0402a970fcd96c8ed01be, State: Running, Role: LEADER
I20250905 02:22:28.598181 3944 consensus_queue.cc:237] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.601019 3639 catalog_manager.cc:5582] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 381c7bc182e94ca9a069646bac9236fd (127.31.48.130). New cstate: current_term: 1 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:28.605171 3637 catalog_manager.cc:5582] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be reported cstate change: term changed from 0 to 1, leader changed from <none> to 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129). New cstate: current_term: 1 leader_uuid: "27a038d5acd0402a970fcd96c8ed01be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:28.716427 3943 raft_consensus.cc:491] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:28.716943 3943 raft_consensus.cc:513] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.718824 3943 leader_election.cc:290] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291), 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:28.728888 3742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485c44a581a941ad9d73b6b68057a359" candidate_uuid: "a384e9e80f10425c850acd9d8d61b45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a038d5acd0402a970fcd96c8ed01be" is_pre_election: true
I20250905 02:22:28.729007 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485c44a581a941ad9d73b6b68057a359" candidate_uuid: "a384e9e80f10425c850acd9d8d61b45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd" is_pre_election: true
I20250905 02:22:28.729504 3742 raft_consensus.cc:2466] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a384e9e80f10425c850acd9d8d61b45f in term 0.
I20250905 02:22:28.729560 3817 raft_consensus.cc:2466] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a384e9e80f10425c850acd9d8d61b45f in term 0.
I20250905 02:22:28.730583 3854 leader_election.cc:304] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27a038d5acd0402a970fcd96c8ed01be, a384e9e80f10425c850acd9d8d61b45f; no voters:
I20250905 02:22:28.731235 3943 raft_consensus.cc:2802] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:28.731487 3943 raft_consensus.cc:491] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:28.731695 3943 raft_consensus.cc:3058] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.736073 3943 raft_consensus.cc:513] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:28.737416 3943 leader_election.cc:290] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 1 election: Requested vote from peers 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291), 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:28.738140 3742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485c44a581a941ad9d73b6b68057a359" candidate_uuid: "a384e9e80f10425c850acd9d8d61b45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27a038d5acd0402a970fcd96c8ed01be"
I20250905 02:22:28.738350 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "485c44a581a941ad9d73b6b68057a359" candidate_uuid: "a384e9e80f10425c850acd9d8d61b45f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd"
I20250905 02:22:28.738750 3742 raft_consensus.cc:3058] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.738873 3817 raft_consensus.cc:3058] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:28.745374 3817 raft_consensus.cc:2466] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a384e9e80f10425c850acd9d8d61b45f in term 1.
I20250905 02:22:28.745378 3742 raft_consensus.cc:2466] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a384e9e80f10425c850acd9d8d61b45f in term 1.
I20250905 02:22:28.746723 3856 leader_election.cc:304] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381c7bc182e94ca9a069646bac9236fd, a384e9e80f10425c850acd9d8d61b45f; no voters:
I20250905 02:22:28.747417 3943 raft_consensus.cc:2802] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:28.748482 3943 raft_consensus.cc:695] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 LEADER]: Becoming Leader. State: Replica: a384e9e80f10425c850acd9d8d61b45f, State: Running, Role: LEADER
I20250905 02:22:28.749562 3943 consensus_queue.cc:237] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
W20250905 02:22:28.755681 3688 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 02:22:28.756798 3637 catalog_manager.cc:5582] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: term changed from 0 to 1, leader changed from <none> to a384e9e80f10425c850acd9d8d61b45f (127.31.48.131). New cstate: current_term: 1 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:28.772735 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:22:28.790627 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:28.791365 31938 tablet_replica.cc:331] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: stopping tablet replica
I20250905 02:22:28.791905 31938 raft_consensus.cc:2241] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:28.792562 31938 pending_rounds.cc:70] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: Trying to abort 1 pending ops.
I20250905 02:22:28.792747 31938 pending_rounds.cc:77] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831533465763840 op_type: NO_OP noop_request { }
I20250905 02:22:28.793056 31938 raft_consensus.cc:2887] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:28.793318 31938 raft_consensus.cc:2270] T be7336e3b2e14e5ca2225619988bd6d6 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:28.795429 31938 tablet_replica.cc:331] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be: stopping tablet replica
I20250905 02:22:28.795930 31938 raft_consensus.cc:2241] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:28.796286 31938 raft_consensus.cc:2270] T 556a3d6954084dc481464d12488767f1 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:28.798038 31938 tablet_replica.cc:331] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: stopping tablet replica
I20250905 02:22:28.798513 31938 raft_consensus.cc:2241] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:28.799031 31938 pending_rounds.cc:70] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: Trying to abort 1 pending ops.
I20250905 02:22:28.799196 31938 pending_rounds.cc:77] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831533331271680 op_type: NO_OP noop_request { }
I20250905 02:22:28.799455 31938 raft_consensus.cc:2887] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:28.799702 31938 raft_consensus.cc:2270] T 78fde155a8eb49e9898595529eb6cce0 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:28.801649 31938 tablet_replica.cc:331] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be: stopping tablet replica
I20250905 02:22:28.802078 31938 raft_consensus.cc:2241] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:28.802393 31938 raft_consensus.cc:2270] T 485c44a581a941ad9d73b6b68057a359 P 27a038d5acd0402a970fcd96c8ed01be [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:28.824141 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
W20250905 02:22:29.125568 3780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20250905 02:22:29.128304 3780 consensus_peers.cc:489] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:29.164981 3969 consensus_queue.cc:1035] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:22:29.244735 3854 consensus_peers.cc:489] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:29.342257 3943 consensus_queue.cc:1035] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:29.696748 3969 consensus_queue.cc:579] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.103s)
W20250905 02:22:29.704474 3637 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 556a3d6954084dc481464d12488767f1 with cas_config_opid_index -1: no extra replica candidate found for tablet 556a3d6954084dc481464d12488767f1 (table test-workload [id=645d4373ef35443ea2557f2125fa1b67]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250905 02:22:29.757629 3688 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 27a038d5acd0402a970fcd96c8ed01be not available for placement
I20250905 02:22:29.795856 3980 consensus_queue.cc:579] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.045s)
I20250905 02:22:29.836280 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:29.841925 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:29.843240 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:29.844761 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:29.844892 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:29.846050 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:29.846246 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:29.846386 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038949846374 us; error 0 us; skew 500 ppm
I20250905 02:22:29.846813 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:29.849165 31938 webserver.cc:480] Webserver started at http://127.31.48.132:40201/ using document root <none> and password file <none>
I20250905 02:22:29.849583 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:29.849781 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:29.850055 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:29.851085 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "9b5943edc54a48159391b031966dbeb5"
format_stamp: "Formatted at 2025-09-05 02:22:29 on dist-test-slave-q32q"
I20250905 02:22:29.855810 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:22:29.858983 3995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:29.859712 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:22:29.859998 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "9b5943edc54a48159391b031966dbeb5"
format_stamp: "Formatted at 2025-09-05 02:22:29 on dist-test-slave-q32q"
I20250905 02:22:29.860261 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.NoRebalancingIfReplicasRecovering.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:29.872931 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:29.873876 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:29.875239 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:29.877353 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:29.877560 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:29.877812 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:29.877985 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:29.915979 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:39039
I20250905 02:22:29.916065 4057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:39039 every 8 connection(s)
I20250905 02:22:29.930094 4058 heartbeater.cc:344] Connected to a master server at 127.31.48.190:36911
I20250905 02:22:29.930466 4058 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:29.931178 4058 heartbeater.cc:507] Master 127.31.48.190:36911 requested a full tablet report, sending...
I20250905 02:22:29.932991 3637 ts_manager.cc:194] Registered new tserver with Master: 9b5943edc54a48159391b031966dbeb5 (127.31.48.132:39039)
I20250905 02:22:29.934417 3637 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52154
I20250905 02:22:29.985913 3890 consensus_queue.cc:237] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:29.991925 3817 raft_consensus.cc:1273] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:22:29.993278 3943 consensus_queue.cc:1035] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:22:29.996872 3854 consensus_peers.cc:489] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:29.999539 3980 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
W20250905 02:22:30.002324 3857 consensus_peers.cc:489] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f -> Peer 9b5943edc54a48159391b031966dbeb5 (127.31.48.132:39039): Couldn't send request to peer 9b5943edc54a48159391b031966dbeb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 485c44a581a941ad9d73b6b68057a359. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:30.002146 3817 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:30.008241 3627 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 485c44a581a941ad9d73b6b68057a359 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250905 02:22:30.011956 3637 catalog_manager.cc:5582] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:30.049364 3817 consensus_queue.cc:237] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:30.056993 3890 raft_consensus.cc:1273] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 02:22:30.057583 3780 consensus_peers.cc:489] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:30.058349 3982 consensus_queue.cc:1035] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:22:30.063975 3783 consensus_peers.cc:489] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd -> Peer 9b5943edc54a48159391b031966dbeb5 (127.31.48.132:39039): Couldn't send request to peer 9b5943edc54a48159391b031966dbeb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 556a3d6954084dc481464d12488767f1. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:30.065016 3969 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:30.066690 3890 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:30.071712 3626 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 556a3d6954084dc481464d12488767f1 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250905 02:22:30.076117 3639 catalog_manager.cc:5582] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd reported cstate change: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:30.381817 4069 ts_tablet_manager.cc:927] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: Initiating tablet copy from peer a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:30.383396 4069 tablet_copy_client.cc:323] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.131:39369
I20250905 02:22:30.391924 3901 tablet_copy_service.cc:140] P a384e9e80f10425c850acd9d8d61b45f: Received BeginTabletCopySession request for tablet 485c44a581a941ad9d73b6b68057a359 from peer 9b5943edc54a48159391b031966dbeb5 ({username='slave'} at 127.0.0.1:56790)
I20250905 02:22:30.392336 3901 tablet_copy_service.cc:161] P a384e9e80f10425c850acd9d8d61b45f: Beginning new tablet copy session on tablet 485c44a581a941ad9d73b6b68057a359 from peer 9b5943edc54a48159391b031966dbeb5 at {username='slave'} at 127.0.0.1:56790: session id = 9b5943edc54a48159391b031966dbeb5-485c44a581a941ad9d73b6b68057a359
I20250905 02:22:30.397068 3901 tablet_copy_source_session.cc:215] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:30.399530 4069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 485c44a581a941ad9d73b6b68057a359. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:30.410036 4069 tablet_copy_client.cc:806] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:30.410578 4069 tablet_copy_client.cc:670] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:30.413699 4069 tablet_copy_client.cc:538] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:30.419683 4069 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: Bootstrap starting.
I20250905 02:22:30.434221 4069 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:30.434857 4069 tablet_bootstrap.cc:492] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: Bootstrap complete.
I20250905 02:22:30.435379 4069 ts_tablet_manager.cc:1397] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20250905 02:22:30.437461 4069 raft_consensus.cc:357] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:30.437999 4069 raft_consensus.cc:738] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b5943edc54a48159391b031966dbeb5, State: Initialized, Role: LEARNER
I20250905 02:22:30.438434 4069 consensus_queue.cc:260] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:30.440037 4058 heartbeater.cc:499] Master 127.31.48.190:36911 was elected leader, sending a full tablet report...
I20250905 02:22:30.440338 4069 ts_tablet_manager.cc:1428] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250905 02:22:30.441855 3901 tablet_copy_service.cc:342] P a384e9e80f10425c850acd9d8d61b45f: Request end of tablet copy session 9b5943edc54a48159391b031966dbeb5-485c44a581a941ad9d73b6b68057a359 received from {username='slave'} at 127.0.0.1:56790
I20250905 02:22:30.442204 3901 tablet_copy_service.cc:434] P a384e9e80f10425c850acd9d8d61b45f: ending tablet copy session 9b5943edc54a48159391b031966dbeb5-485c44a581a941ad9d73b6b68057a359 on tablet 485c44a581a941ad9d73b6b68057a359 with peer 9b5943edc54a48159391b031966dbeb5
I20250905 02:22:30.531536 4069 ts_tablet_manager.cc:927] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: Initiating tablet copy from peer 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:30.532812 4069 tablet_copy_client.cc:323] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.130:39449
I20250905 02:22:30.535264 3982 raft_consensus.cc:491] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:30.535679 3982 raft_consensus.cc:513] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:30.537289 3982 leader_election.cc:290] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291), a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:30.538371 3890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78fde155a8eb49e9898595529eb6cce0" candidate_uuid: "381c7bc182e94ca9a069646bac9236fd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f" is_pre_election: true
I20250905 02:22:30.539028 3890 raft_consensus.cc:2466] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 381c7bc182e94ca9a069646bac9236fd in term 1.
I20250905 02:22:30.540339 3783 leader_election.cc:304] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381c7bc182e94ca9a069646bac9236fd, a384e9e80f10425c850acd9d8d61b45f; no voters:
W20250905 02:22:30.541064 3780 leader_election.cc:336] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:30.541237 3982 raft_consensus.cc:2802] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:30.541689 3982 raft_consensus.cc:491] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:30.542037 3982 raft_consensus.cc:3058] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:30.543823 3827 tablet_copy_service.cc:140] P 381c7bc182e94ca9a069646bac9236fd: Received BeginTabletCopySession request for tablet 556a3d6954084dc481464d12488767f1 from peer 9b5943edc54a48159391b031966dbeb5 ({username='slave'} at 127.0.0.1:48186)
I20250905 02:22:30.544291 3827 tablet_copy_service.cc:161] P 381c7bc182e94ca9a069646bac9236fd: Beginning new tablet copy session on tablet 556a3d6954084dc481464d12488767f1 from peer 9b5943edc54a48159391b031966dbeb5 at {username='slave'} at 127.0.0.1:48186: session id = 9b5943edc54a48159391b031966dbeb5-556a3d6954084dc481464d12488767f1
I20250905 02:22:30.547554 3982 raft_consensus.cc:513] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:30.549177 3982 leader_election.cc:290] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 2 election: Requested vote from peers 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291), a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:30.550166 3890 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78fde155a8eb49e9898595529eb6cce0" candidate_uuid: "381c7bc182e94ca9a069646bac9236fd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a384e9e80f10425c850acd9d8d61b45f"
I20250905 02:22:30.549525 3827 tablet_copy_source_session.cc:215] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:30.550638 3890 raft_consensus.cc:3058] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Advancing to term 2
W20250905 02:22:30.552704 3780 leader_election.cc:336] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:30.553529 4069 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 556a3d6954084dc481464d12488767f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:30.555928 3890 raft_consensus.cc:2466] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 381c7bc182e94ca9a069646bac9236fd in term 2.
I20250905 02:22:30.556877 3783 leader_election.cc:304] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 381c7bc182e94ca9a069646bac9236fd, a384e9e80f10425c850acd9d8d61b45f; no voters: 27a038d5acd0402a970fcd96c8ed01be
I20250905 02:22:30.557746 3982 raft_consensus.cc:2802] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:30.558334 3982 raft_consensus.cc:695] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 LEADER]: Becoming Leader. State: Replica: 381c7bc182e94ca9a069646bac9236fd, State: Running, Role: LEADER
I20250905 02:22:30.558986 3982 consensus_queue.cc:237] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:30.566145 4069 tablet_copy_client.cc:806] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:30.566797 4069 tablet_copy_client.cc:670] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:30.566231 3637 catalog_manager.cc:5582] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd reported cstate change: term changed from 1 to 2, leader changed from 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) to 381c7bc182e94ca9a069646bac9236fd (127.31.48.130). New cstate: current_term: 2 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:30.570148 4069 tablet_copy_client.cc:538] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:30.577575 4069 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: Bootstrap starting.
I20250905 02:22:30.591195 4069 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:30.591851 4069 tablet_bootstrap.cc:492] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: Bootstrap complete.
I20250905 02:22:30.592356 4069 ts_tablet_manager.cc:1397] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20250905 02:22:30.594159 4069 raft_consensus.cc:357] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:30.594703 4069 raft_consensus.cc:738] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9b5943edc54a48159391b031966dbeb5, State: Initialized, Role: LEARNER
I20250905 02:22:30.595139 4069 consensus_queue.cc:260] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:30.596843 4069 ts_tablet_manager.cc:1428] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:30.598217 3827 tablet_copy_service.cc:342] P 381c7bc182e94ca9a069646bac9236fd: Request end of tablet copy session 9b5943edc54a48159391b031966dbeb5-556a3d6954084dc481464d12488767f1 received from {username='slave'} at 127.0.0.1:48186
I20250905 02:22:30.598523 3827 tablet_copy_service.cc:434] P 381c7bc182e94ca9a069646bac9236fd: ending tablet copy session 9b5943edc54a48159391b031966dbeb5-556a3d6954084dc481464d12488767f1 on tablet 556a3d6954084dc481464d12488767f1 with peer 9b5943edc54a48159391b031966dbeb5
I20250905 02:22:30.604839 3980 raft_consensus.cc:491] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:30.605296 3980 raft_consensus.cc:513] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:30.606763 3980 leader_election.cc:290] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291), 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:30.608007 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7336e3b2e14e5ca2225619988bd6d6" candidate_uuid: "a384e9e80f10425c850acd9d8d61b45f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd" is_pre_election: true
I20250905 02:22:30.608690 3817 raft_consensus.cc:2466] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a384e9e80f10425c850acd9d8d61b45f in term 1.
I20250905 02:22:30.609663 3856 leader_election.cc:304] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 381c7bc182e94ca9a069646bac9236fd, a384e9e80f10425c850acd9d8d61b45f; no voters:
W20250905 02:22:30.610121 3854 leader_election.cc:336] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:30.610296 3980 raft_consensus.cc:2802] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:30.610651 3980 raft_consensus.cc:491] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:30.610900 3980 raft_consensus.cc:3058] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:30.615598 3980 raft_consensus.cc:513] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:30.617166 3980 leader_election.cc:290] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 2 election: Requested vote from peers 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291), 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:30.618278 3817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be7336e3b2e14e5ca2225619988bd6d6" candidate_uuid: "a384e9e80f10425c850acd9d8d61b45f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "381c7bc182e94ca9a069646bac9236fd"
I20250905 02:22:30.618889 3817 raft_consensus.cc:3058] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Advancing to term 2
W20250905 02:22:30.620546 3854 leader_election.cc:336] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:30.623960 3817 raft_consensus.cc:2466] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a384e9e80f10425c850acd9d8d61b45f in term 2.
I20250905 02:22:30.624943 3856 leader_election.cc:304] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 381c7bc182e94ca9a069646bac9236fd, a384e9e80f10425c850acd9d8d61b45f; no voters: 27a038d5acd0402a970fcd96c8ed01be
I20250905 02:22:30.625643 3980 raft_consensus.cc:2802] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:30.626449 3980 raft_consensus.cc:695] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 LEADER]: Becoming Leader. State: Replica: a384e9e80f10425c850acd9d8d61b45f, State: Running, Role: LEADER
I20250905 02:22:30.627085 3980 consensus_queue.cc:237] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } }
I20250905 02:22:30.633664 3637 catalog_manager.cc:5582] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: term changed from 1 to 2, leader changed from 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) to a384e9e80f10425c850acd9d8d61b45f (127.31.48.131). New cstate: current_term: 2 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } }
W20250905 02:22:30.759552 3688 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 27a038d5acd0402a970fcd96c8ed01be not available for placement
I20250905 02:22:30.966493 4033 raft_consensus.cc:1215] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20250905 02:22:31.007056 3780 consensus_peers.cc:489] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:31.014863 3817 raft_consensus.cc:1273] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250905 02:22:31.016562 3980 consensus_queue.cc:1035] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 02:22:31.031953 3854 consensus_peers.cc:489] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:31.135219 3982 consensus_queue.cc:579] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.084s)
I20250905 02:22:31.173327 4033 raft_consensus.cc:1215] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 02:22:31.174744 3890 raft_consensus.cc:1273] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250905 02:22:31.176007 3969 consensus_queue.cc:1035] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:31.430778 4078 consensus_queue.cc:579] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.443s)
I20250905 02:22:31.501454 4080 raft_consensus.cc:1062] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: attempting to promote NON_VOTER 9b5943edc54a48159391b031966dbeb5 to VOTER
I20250905 02:22:31.503892 4080 consensus_queue.cc:237] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:31.510210 3817 raft_consensus.cc:1273] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:31.510224 4033 raft_consensus.cc:1273] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:31.511667 4078 consensus_queue.cc:1035] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:31.512509 4080 consensus_queue.cc:1035] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250905 02:22:31.513831 3854 consensus_peers.cc:489] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:31.522554 4078 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.524082 3817 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.533775 4033 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.536330 3639 catalog_manager.cc:5582] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 02:22:31.555692 3890 consensus_queue.cc:237] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:31.560289 4033 raft_consensus.cc:1273] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:31.560714 3817 raft_consensus.cc:1273] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:31.561888 4078 consensus_queue.cc:1035] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:31.562703 4080 consensus_queue.cc:1035] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:31.569304 4087 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.571033 3817 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.571033 4033 raft_consensus.cc:2953] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.577953 3627 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 485c44a581a941ad9d73b6b68057a359 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:31.581532 3640 catalog_manager.cc:5582] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New cstate: current_term: 1 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 02:22:31.598734 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 485c44a581a941ad9d73b6b68057a359: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:31.639586 4087 consensus_queue.cc:579] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.012s)
I20250905 02:22:31.649407 3890 consensus_queue.cc:237] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:31.654569 3817 raft_consensus.cc:1273] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250905 02:22:31.655112 3857 consensus_peers.cc:489] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f -> Peer 9b5943edc54a48159391b031966dbeb5 (127.31.48.132:39039): Couldn't send request to peer 9b5943edc54a48159391b031966dbeb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be7336e3b2e14e5ca2225619988bd6d6. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:31.656278 4087 consensus_queue.cc:1035] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:22:31.657614 3854 consensus_peers.cc:489] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:31.662321 4078 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:31.663873 3817 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:31.671747 3627 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet be7336e3b2e14e5ca2225619988bd6d6 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250905 02:22:31.675751 3637 catalog_manager.cc:5582] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:31.697374 3969 raft_consensus.cc:1062] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: attempting to promote NON_VOTER 9b5943edc54a48159391b031966dbeb5 to VOTER
I20250905 02:22:31.699518 3969 consensus_queue.cc:237] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:31.706045 4033 raft_consensus.cc:1273] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 LEARNER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:22:31.707536 4097 consensus_queue.cc:1035] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:31.708016 3890 raft_consensus.cc:1273] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20250905 02:22:31.708420 3780 consensus_peers.cc:489] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:31.709678 4095 consensus_queue.cc:1035] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:31.717443 4097 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.719044 3890 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.727020 4033 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.729774 3639 catalog_manager.cc:5582] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd reported cstate change: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 02:22:31.744249 3817 consensus_queue.cc:237] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:31.748759 4033 raft_consensus.cc:1273] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:31.748986 3890 raft_consensus.cc:1273] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20250905 02:22:31.750058 4097 consensus_queue.cc:1035] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:31.750633 3969 consensus_queue.cc:1035] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:31.756682 4095 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.758152 3890 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:31.758464 4033 raft_consensus.cc:2953] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
W20250905 02:22:31.762610 3688 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 27a038d5acd0402a970fcd96c8ed01be not available for placement
I20250905 02:22:31.766781 3626 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 556a3d6954084dc481464d12488767f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:31.771587 3640 catalog_manager.cc:5582] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New cstate: current_term: 1 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:32.033953 3969 consensus_queue.cc:579] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.474s)
I20250905 02:22:32.045130 3817 consensus_queue.cc:237] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
W20250905 02:22:32.050804 3783 consensus_peers.cc:489] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd -> Peer 9b5943edc54a48159391b031966dbeb5 (127.31.48.132:39039): Couldn't send request to peer 9b5943edc54a48159391b031966dbeb5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78fde155a8eb49e9898595529eb6cce0. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:32.051963 3890 raft_consensus.cc:1273] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250905 02:22:32.053067 3780 consensus_peers.cc:489] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:32.053254 4097 consensus_queue.cc:1035] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:32.058060 4095 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:32.059587 3890 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
I20250905 02:22:32.066998 3626 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 78fde155a8eb49e9898595529eb6cce0 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250905 02:22:32.069716 3671 debug-util.cc:398] Leaking SignalData structure 0x7b0800181c00 after lost signal to thread 31941
W20250905 02:22:32.071224 3671 debug-util.cc:398] Leaking SignalData structure 0x7b080018a1e0 after lost signal to thread 3674
W20250905 02:22:32.072824 3671 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4b20 after lost signal to thread 3841
I20250905 02:22:32.072856 3639 catalog_manager.cc:5582] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f reported cstate change: config changed from index -1 to 2, NON_VOTER 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } } }
W20250905 02:22:32.074263 3671 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4dc0 after lost signal to thread 3915
W20250905 02:22:32.075791 3671 debug-util.cc:398] Leaking SignalData structure 0x7b08001d0000 after lost signal to thread 4057
I20250905 02:22:32.128314 4101 ts_tablet_manager.cc:927] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: Initiating tablet copy from peer a384e9e80f10425c850acd9d8d61b45f (127.31.48.131:39369)
I20250905 02:22:32.130152 4101 tablet_copy_client.cc:323] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.131:39369
I20250905 02:22:32.131783 3901 tablet_copy_service.cc:140] P a384e9e80f10425c850acd9d8d61b45f: Received BeginTabletCopySession request for tablet be7336e3b2e14e5ca2225619988bd6d6 from peer 9b5943edc54a48159391b031966dbeb5 ({username='slave'} at 127.0.0.1:56790)
I20250905 02:22:32.132198 3901 tablet_copy_service.cc:161] P a384e9e80f10425c850acd9d8d61b45f: Beginning new tablet copy session on tablet be7336e3b2e14e5ca2225619988bd6d6 from peer 9b5943edc54a48159391b031966dbeb5 at {username='slave'} at 127.0.0.1:56790: session id = 9b5943edc54a48159391b031966dbeb5-be7336e3b2e14e5ca2225619988bd6d6
I20250905 02:22:32.137724 3901 tablet_copy_source_session.cc:215] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:32.140450 4101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be7336e3b2e14e5ca2225619988bd6d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:32.155568 4101 tablet_copy_client.cc:806] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:32.156185 4101 tablet_copy_client.cc:670] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:32.161273 4101 tablet_copy_client.cc:538] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:32.168062 4101 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: Bootstrap starting.
I20250905 02:22:32.185923 4101 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:32.186658 4101 tablet_bootstrap.cc:492] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: Bootstrap complete.
I20250905 02:22:32.187211 4101 ts_tablet_manager.cc:1397] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.004s
I20250905 02:22:32.189620 4101 raft_consensus.cc:357] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:32.190441 4101 raft_consensus.cc:738] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9b5943edc54a48159391b031966dbeb5, State: Initialized, Role: LEARNER
I20250905 02:22:32.191076 4101 consensus_queue.cc:260] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:32.193696 4101 ts_tablet_manager.cc:1428] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250905 02:22:32.195365 3901 tablet_copy_service.cc:342] P a384e9e80f10425c850acd9d8d61b45f: Request end of tablet copy session 9b5943edc54a48159391b031966dbeb5-be7336e3b2e14e5ca2225619988bd6d6 received from {username='slave'} at 127.0.0.1:56790
I20250905 02:22:32.195894 3901 tablet_copy_service.cc:434] P a384e9e80f10425c850acd9d8d61b45f: ending tablet copy session 9b5943edc54a48159391b031966dbeb5-be7336e3b2e14e5ca2225619988bd6d6 on tablet be7336e3b2e14e5ca2225619988bd6d6 with peer 9b5943edc54a48159391b031966dbeb5
I20250905 02:22:32.490037 4101 ts_tablet_manager.cc:927] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: Initiating tablet copy from peer 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449)
I20250905 02:22:32.491719 4101 tablet_copy_client.cc:323] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.130:39449
I20250905 02:22:32.493690 3827 tablet_copy_service.cc:140] P 381c7bc182e94ca9a069646bac9236fd: Received BeginTabletCopySession request for tablet 78fde155a8eb49e9898595529eb6cce0 from peer 9b5943edc54a48159391b031966dbeb5 ({username='slave'} at 127.0.0.1:48186)
I20250905 02:22:32.494120 3827 tablet_copy_service.cc:161] P 381c7bc182e94ca9a069646bac9236fd: Beginning new tablet copy session on tablet 78fde155a8eb49e9898595529eb6cce0 from peer 9b5943edc54a48159391b031966dbeb5 at {username='slave'} at 127.0.0.1:48186: session id = 9b5943edc54a48159391b031966dbeb5-78fde155a8eb49e9898595529eb6cce0
I20250905 02:22:32.499456 3827 tablet_copy_source_session.cc:215] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:32.502522 4101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78fde155a8eb49e9898595529eb6cce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:32.520716 4101 tablet_copy_client.cc:806] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:32.521391 4101 tablet_copy_client.cc:670] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:32.524971 4101 tablet_copy_client.cc:538] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:32.535759 4101 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: Bootstrap starting.
I20250905 02:22:32.595788 4101 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:32.596822 4101 tablet_bootstrap.cc:492] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: Bootstrap complete.
I20250905 02:22:32.597647 4101 ts_tablet_manager.cc:1397] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: Time spent bootstrapping tablet: real 0.062s user 0.024s sys 0.007s
I20250905 02:22:32.600955 4101 raft_consensus.cc:357] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:32.602005 4101 raft_consensus.cc:738] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9b5943edc54a48159391b031966dbeb5, State: Initialized, Role: LEARNER
I20250905 02:22:32.602825 4101 consensus_queue.cc:260] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: true } }
I20250905 02:22:32.607136 4101 ts_tablet_manager.cc:1428] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250905 02:22:32.702052 4033 raft_consensus.cc:1215] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250905 02:22:32.703179 4108 consensus_queue.cc:579] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.053s)
I20250905 02:22:32.788640 3827 tablet_copy_service.cc:342] P 381c7bc182e94ca9a069646bac9236fd: Request end of tablet copy session 9b5943edc54a48159391b031966dbeb5-78fde155a8eb49e9898595529eb6cce0 received from {username='slave'} at 127.0.0.1:48186
I20250905 02:22:32.789037 3827 tablet_copy_service.cc:434] P 381c7bc182e94ca9a069646bac9236fd: ending tablet copy session 9b5943edc54a48159391b031966dbeb5-78fde155a8eb49e9898595529eb6cce0 on tablet 78fde155a8eb49e9898595529eb6cce0 with peer 9b5943edc54a48159391b031966dbeb5
W20250905 02:22:32.799192 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 485c44a581a941ad9d73b6b68057a359: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
W20250905 02:22:32.800129 3688 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 27a038d5acd0402a970fcd96c8ed01be not available for placement
I20250905 02:22:33.048677 4095 consensus_queue.cc:579] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Leader has been unable to successfully communicate with peer 27a038d5acd0402a970fcd96c8ed01be for more than 1 seconds (1.002s)
I20250905 02:22:33.104844 4033 raft_consensus.cc:1215] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20250905 02:22:33.236804 4080 raft_consensus.cc:1062] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: attempting to promote NON_VOTER 9b5943edc54a48159391b031966dbeb5 to VOTER
I20250905 02:22:33.239455 4080 consensus_queue.cc:237] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:33.246636 3817 raft_consensus.cc:1273] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:22:33.246678 4033 raft_consensus.cc:1273] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:22:33.248234 4108 consensus_queue.cc:1035] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:33.248890 4080 consensus_queue.cc:1035] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
W20250905 02:22:33.250818 3854 consensus_peers.cc:489] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:33.259171 4080 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.260628 4033 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.271576 3639 catalog_manager.cc:5582] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 reported cstate change: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.273079 3817 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.296609 3890 consensus_queue.cc:237] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:33.301563 3817 raft_consensus.cc:1273] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:22:33.301929 4033 raft_consensus.cc:1273] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Refusing update from remote peer a384e9e80f10425c850acd9d8d61b45f: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:22:33.303117 4080 consensus_queue.cc:1035] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:33.303676 4110 consensus_queue.cc:1035] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:33.309624 4109 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.311146 3817 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.311146 4033 raft_consensus.cc:2953] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.321409 3627 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet be7336e3b2e14e5ca2225619988bd6d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:33.325886 3637 catalog_manager.cc:5582] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd reported cstate change: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New cstate: current_term: 2 leader_uuid: "a384e9e80f10425c850acd9d8d61b45f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
W20250905 02:22:33.619390 4054 debug-util.cc:398] Leaking SignalData structure 0x7b080008ed60 after lost signal to thread 31941
W20250905 02:22:33.620299 4054 debug-util.cc:398] Leaking SignalData structure 0x7b08001949a0 after lost signal to thread 3674
W20250905 02:22:33.621317 4054 debug-util.cc:398] Leaking SignalData structure 0x7b080022e660 after lost signal to thread 3841
W20250905 02:22:33.622177 4054 debug-util.cc:398] Leaking SignalData structure 0x7b08000341a0 after lost signal to thread 3915
W20250905 02:22:33.623112 4054 debug-util.cc:398] Leaking SignalData structure 0x7b080009c700 after lost signal to thread 4057
I20250905 02:22:33.674790 4095 raft_consensus.cc:1062] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: attempting to promote NON_VOTER 9b5943edc54a48159391b031966dbeb5 to VOTER
I20250905 02:22:33.677696 4095 consensus_queue.cc:237] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:33.685775 4033 raft_consensus.cc:1273] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 LEARNER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:22:33.690542 3890 raft_consensus.cc:1273] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:22:33.690488 4112 consensus_queue.cc:1035] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20250905 02:22:33.700152 4112 consensus_queue.cc:1035] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250905 02:22:33.705190 3780 consensus_peers.cc:489] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd -> Peer 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): Couldn't send request to peer 27a038d5acd0402a970fcd96c8ed01be. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:33.728710 4112 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.730602 4032 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.747196 3639 catalog_manager.cc:5582] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 reported cstate change: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.748274 3890 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 9b5943edc54a48159391b031966dbeb5 (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "27a038d5acd0402a970fcd96c8ed01be" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 36291 } } peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.771939 3817 consensus_queue.cc:237] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } }
I20250905 02:22:33.776757 3890 raft_consensus.cc:1273] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:22:33.776757 4032 raft_consensus.cc:1273] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Refusing update from remote peer 381c7bc182e94ca9a069646bac9236fd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:22:33.778544 4123 consensus_queue.cc:1035] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20250905 02:22:33.779314 4095 consensus_queue.cc:1035] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:22:33.787029 4112 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.788954 4032 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
I20250905 02:22:33.798554 3626 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 78fde155a8eb49e9898595529eb6cce0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:22:33.790987 3890 raft_consensus.cc:2953] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } } }
W20250905 02:22:33.805825 3688 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 27a038d5acd0402a970fcd96c8ed01be not available for placement
I20250905 02:22:33.809295 3639 catalog_manager.cc:5582] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd reported cstate change: config changed from index 3 to 4, VOTER 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129) evicted. New cstate: current_term: 2 leader_uuid: "381c7bc182e94ca9a069646bac9236fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "381c7bc182e94ca9a069646bac9236fd" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 39449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a384e9e80f10425c850acd9d8d61b45f" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 39369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b5943edc54a48159391b031966dbeb5" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 39039 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 02:22:33.844564 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 78fde155a8eb49e9898595529eb6cce0: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
W20250905 02:22:34.185269 3624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20250905 02:22:35.035804 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 78fde155a8eb49e9898595529eb6cce0: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
W20250905 02:22:35.921566 3857 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.48.132:39039, user_credentials={real_user=slave}} blocked reactor thread for 66443.6us
W20250905 02:22:36.194427 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 485c44a581a941ad9d73b6b68057a359: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
W20250905 02:22:37.013793 3671 debug-util.cc:398] Leaking SignalData structure 0x7b0800093480 after lost signal to thread 31941
W20250905 02:22:37.014937 3671 debug-util.cc:398] Leaking SignalData structure 0x7b08001cf900 after lost signal to thread 3674
W20250905 02:22:37.016234 3671 debug-util.cc:398] Leaking SignalData structure 0x7b0800216220 after lost signal to thread 3841
W20250905 02:22:37.017409 3671 debug-util.cc:398] Leaking SignalData structure 0x7b08000d0040 after lost signal to thread 3915
W20250905 02:22:37.018751 3671 debug-util.cc:398] Leaking SignalData structure 0x7b0800125700 after lost signal to thread 4057
I20250905 02:22:37.541610 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:22:37.563730 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:37.564621 31938 tablet_replica.cc:331] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd: stopping tablet replica
I20250905 02:22:37.565300 31938 raft_consensus.cc:2241] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:22:37.566102 31938 raft_consensus.cc:2270] T 78fde155a8eb49e9898595529eb6cce0 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.567989 31938 tablet_replica.cc:331] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd: stopping tablet replica
I20250905 02:22:37.568522 31938 raft_consensus.cc:2241] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.569089 31938 raft_consensus.cc:2270] T be7336e3b2e14e5ca2225619988bd6d6 P 381c7bc182e94ca9a069646bac9236fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.570855 31938 tablet_replica.cc:331] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd: stopping tablet replica
W20250905 02:22:37.565294 3856 consensus_peers.cc:489] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f -> Peer 381c7bc182e94ca9a069646bac9236fd (127.31.48.130:39449): Couldn't send request to peer 381c7bc182e94ca9a069646bac9236fd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:37.571574 31938 raft_consensus.cc:2241] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:37.572263 31938 raft_consensus.cc:2270] T 556a3d6954084dc481464d12488767f1 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.574107 31938 tablet_replica.cc:331] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd: stopping tablet replica
I20250905 02:22:37.574522 31938 raft_consensus.cc:2241] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.574998 31938 raft_consensus.cc:2270] T 485c44a581a941ad9d73b6b68057a359 P 381c7bc182e94ca9a069646bac9236fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.598392 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:22:37.612720 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:22:37.633646 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:37.634238 31938 tablet_replica.cc:331] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f: stopping tablet replica
I20250905 02:22:37.634748 31938 raft_consensus.cc:2241] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:37.635435 31938 raft_consensus.cc:2270] T 485c44a581a941ad9d73b6b68057a359 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.637331 31938 tablet_replica.cc:331] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f: stopping tablet replica
I20250905 02:22:37.637794 31938 raft_consensus.cc:2241] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.638283 31938 raft_consensus.cc:2270] T 78fde155a8eb49e9898595529eb6cce0 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.639907 31938 tablet_replica.cc:331] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f: stopping tablet replica
I20250905 02:22:37.640514 31938 raft_consensus.cc:2241] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:22:37.641186 31938 raft_consensus.cc:2270] T be7336e3b2e14e5ca2225619988bd6d6 P a384e9e80f10425c850acd9d8d61b45f [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.642910 31938 tablet_replica.cc:331] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f: stopping tablet replica
I20250905 02:22:37.643317 31938 raft_consensus.cc:2241] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.643770 31938 raft_consensus.cc:2270] T 556a3d6954084dc481464d12488767f1 P a384e9e80f10425c850acd9d8d61b45f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.666712 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
W20250905 02:22:37.675861 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet be7336e3b2e14e5ca2225619988bd6d6: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:37.682912 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:22:37.705655 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:37.706249 31938 tablet_replica.cc:331] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5: stopping tablet replica
I20250905 02:22:37.706754 31938 raft_consensus.cc:2241] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.707258 31938 raft_consensus.cc:2270] T 78fde155a8eb49e9898595529eb6cce0 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.708992 31938 tablet_replica.cc:331] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5: stopping tablet replica
I20250905 02:22:37.709443 31938 raft_consensus.cc:2241] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.709887 31938 raft_consensus.cc:2270] T be7336e3b2e14e5ca2225619988bd6d6 P 9b5943edc54a48159391b031966dbeb5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.711405 31938 tablet_replica.cc:331] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5: stopping tablet replica
I20250905 02:22:37.711812 31938 raft_consensus.cc:2241] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.712265 31938 raft_consensus.cc:2270] T 556a3d6954084dc481464d12488767f1 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.713901 31938 tablet_replica.cc:331] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5: stopping tablet replica
I20250905 02:22:37.714290 31938 raft_consensus.cc:2241] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:37.714725 31938 raft_consensus.cc:2270] T 485c44a581a941ad9d73b6b68057a359 P 9b5943edc54a48159391b031966dbeb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:37.756951 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:22:37.769894 31938 master.cc:561] Master@127.31.48.190:36911 shutting down...
W20250905 02:22:40.296305 3624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20250905 02:22:40.299237 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 556a3d6954084dc481464d12488767f1: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
W20250905 02:22:41.805260 3624 catalog_manager.cc:4671] TS 27a038d5acd0402a970fcd96c8ed01be (127.31.48.129:36291): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet be7336e3b2e14e5ca2225619988bd6d6: Network error: Client connection negotiation failed: client connection to 127.31.48.129:36291: connect: Connection refused (error 111)
I20250905 02:22:42.274825 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:42.275821 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:42.276208 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P c863edee63be4b13b7846bc562900dde: stopping tablet replica
W20250905 02:22:42.395102 3671 debug-util.cc:398] Leaking SignalData structure 0x7b080005dbe0 after lost signal to thread 31941
W20250905 02:22:42.395645 3671 debug-util.cc:398] Leaking SignalData structure 0x7b080010f6a0 after lost signal to thread 3674
I20250905 02:22:42.574537 31938 master.cc:583] Master@127.31.48.190:36911 shutdown complete.
[ OK ] AutoRebalancerTest.NoRebalancingIfReplicasRecovering (15999 ms)
[ RUN ] AutoRebalancerTest.TestHandlingFailedTservers
I20250905 02:22:42.621402 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:34765
I20250905 02:22:42.622504 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:42.628001 4146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.628185 4147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.629302 4149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:42.631409 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:42.632239 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:42.632467 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:42.632637 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038962632600 us; error 0 us; skew 500 ppm
I20250905 02:22:42.633183 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:42.635499 31938 webserver.cc:480] Webserver started at http://127.31.48.190:33573/ using document root <none> and password file <none>
I20250905 02:22:42.635963 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:42.636143 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:42.636591 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:42.637838 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "7727fa2a20e84a9d999069929f8b176b"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:42.642552 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20250905 02:22:42.646083 4154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.646849 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250905 02:22:42.647121 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "7727fa2a20e84a9d999069929f8b176b"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:42.647375 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:42.661016 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:42.662207 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:42.699666 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:34765
I20250905 02:22:42.699765 4205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:34765 every 8 connection(s)
I20250905 02:22:42.703668 4206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:42.715835 4206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b: Bootstrap starting.
I20250905 02:22:42.720700 4206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:42.725029 4206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b: No bootstrap required, opened a new log
I20250905 02:22:42.727164 4206 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7727fa2a20e84a9d999069929f8b176b" member_type: VOTER }
I20250905 02:22:42.727576 4206 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:42.727820 4206 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7727fa2a20e84a9d999069929f8b176b, State: Initialized, Role: FOLLOWER
I20250905 02:22:42.728451 4206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7727fa2a20e84a9d999069929f8b176b" member_type: VOTER }
I20250905 02:22:42.728919 4206 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:42.729146 4206 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:42.729404 4206 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:42.734530 4206 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7727fa2a20e84a9d999069929f8b176b" member_type: VOTER }
I20250905 02:22:42.735023 4206 leader_election.cc:304] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7727fa2a20e84a9d999069929f8b176b; no voters:
I20250905 02:22:42.736146 4206 leader_election.cc:290] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:42.736573 4209 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:42.737917 4209 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 1 LEADER]: Becoming Leader. State: Replica: 7727fa2a20e84a9d999069929f8b176b, State: Running, Role: LEADER
I20250905 02:22:42.738615 4209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7727fa2a20e84a9d999069929f8b176b" member_type: VOTER }
I20250905 02:22:42.739109 4206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:42.748327 4211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7727fa2a20e84a9d999069929f8b176b. Latest consensus state: current_term: 1 leader_uuid: "7727fa2a20e84a9d999069929f8b176b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7727fa2a20e84a9d999069929f8b176b" member_type: VOTER } }
I20250905 02:22:42.749037 4211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:42.748730 4210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7727fa2a20e84a9d999069929f8b176b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7727fa2a20e84a9d999069929f8b176b" member_type: VOTER } }
I20250905 02:22:42.749406 4210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:42.750798 4219 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:42.756657 4219 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:42.757768 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:22:42.765185 4219 catalog_manager.cc:1349] Generated new cluster ID: fa6a50e6fe6f40a884e2dad510018bc9
I20250905 02:22:42.765422 4219 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:42.793704 4219 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:42.794993 4219 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:42.812065 4219 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b: Generated new TSK 0
I20250905 02:22:42.812660 4219 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:42.824895 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:42.830916 4227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.832841 4228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.833914 4230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:42.842038 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:42.844142 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:42.844472 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:42.844730 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038962844710 us; error 0 us; skew 500 ppm
I20250905 02:22:42.845436 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:42.848788 31938 webserver.cc:480] Webserver started at http://127.31.48.129:36261/ using document root <none> and password file <none>
I20250905 02:22:42.849579 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:42.849844 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:42.850185 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:42.851589 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "54835050b80544d880704fa9291f1905"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:42.856664 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20250905 02:22:42.859937 4235 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.860679 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:22:42.860961 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "54835050b80544d880704fa9291f1905"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:42.861238 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:42.879333 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:42.880553 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:42.882050 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:42.884397 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:42.884618 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.884860 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:42.885025 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.927477 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:33149
I20250905 02:22:42.927577 4297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:33149 every 8 connection(s)
I20250905 02:22:42.932125 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:42.939190 4302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.940258 4303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:42.943147 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:42.943832 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:42.945215 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:42.945508 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:42.945721 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038962945703 us; error 0 us; skew 500 ppm
I20250905 02:22:42.945978 4298 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:42.946307 4298 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:42.946316 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:42.947218 4298 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:42.949294 4171 ts_manager.cc:194] Registered new tserver with Master: 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:42.949394 31938 webserver.cc:480] Webserver started at http://127.31.48.130:44057/ using document root <none> and password file <none>
I20250905 02:22:42.949999 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:42.950183 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:42.950438 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:42.951229 4171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59728
I20250905 02:22:42.951660 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "6c7070b76dbc426784bf84a6040700d1"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:42.955962 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:22:42.959213 4310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.959879 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:22:42.960175 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "6c7070b76dbc426784bf84a6040700d1"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:42.960464 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:42.979889 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:42.981130 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:42.982839 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:42.986137 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:42.986402 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:42.986675 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:42.986874 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:43.025429 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:36981
I20250905 02:22:43.025534 4372 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:36981 every 8 connection(s)
I20250905 02:22:43.030267 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:43.036762 4376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:43.037261 4377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:43.039062 4379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:43.039650 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:43.040607 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:43.040874 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:43.041100 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038963041080 us; error 0 us; skew 500 ppm
I20250905 02:22:43.041692 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:43.042325 4373 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:43.042765 4373 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:43.043702 4373 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:43.044445 31938 webserver.cc:480] Webserver started at http://127.31.48.131:34057/ using document root <none> and password file <none>
I20250905 02:22:43.045042 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:43.045274 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:43.045595 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:43.045918 4171 ts_manager.cc:194] Registered new tserver with Master: 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
I20250905 02:22:43.046744 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "437e86f2e6c646c59fd56f2315f1f81b"
format_stamp: "Formatted at 2025-09-05 02:22:43 on dist-test-slave-q32q"
I20250905 02:22:43.047269 4171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59742
I20250905 02:22:43.051460 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:22:43.054832 4384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:43.055608 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:22:43.055900 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "437e86f2e6c646c59fd56f2315f1f81b"
format_stamp: "Formatted at 2025-09-05 02:22:43 on dist-test-slave-q32q"
I20250905 02:22:43.056174 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:43.066819 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:43.068003 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:43.069463 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:43.071805 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:43.072005 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:43.072232 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:43.072429 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250905 02:22:43.113370 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:46329
I20250905 02:22:43.113472 4446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:46329 every 8 connection(s)
I20250905 02:22:43.127243 4447 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:43.127588 4447 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:43.128324 4447 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:43.130038 4171 ts_manager.cc:194] Registered new tserver with Master: 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:43.130265 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.01342271s
I20250905 02:22:43.131420 4171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59746
I20250905 02:22:43.953814 4298 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
I20250905 02:22:44.049670 4373 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
I20250905 02:22:44.133975 4447 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
I20250905 02:22:44.163025 31938 test_util.cc:276] Using random seed: -2091859527
I20250905 02:22:44.192546 4171 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59760:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:22:44.194893 4171 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:22:44.246855 4412 tablet_service.cc:1468] Processing CreateTablet for tablet 691922bedd744294a3219a5605970305 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:22:44.247546 4411 tablet_service.cc:1468] Processing CreateTablet for tablet 1745e6d53a45442c91d58026e7c5c0c1 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:22:44.248410 4412 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691922bedd744294a3219a5605970305. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.249868 4337 tablet_service.cc:1468] Processing CreateTablet for tablet 1745e6d53a45442c91d58026e7c5c0c1 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:22:44.251271 4337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745e6d53a45442c91d58026e7c5c0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.250953 4336 tablet_service.cc:1468] Processing CreateTablet for tablet b922492bc53645bf90e75036b5aa3a8a (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:22:44.253614 4336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b922492bc53645bf90e75036b5aa3a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.253460 4410 tablet_service.cc:1468] Processing CreateTablet for tablet b922492bc53645bf90e75036b5aa3a8a (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:22:44.254447 4335 tablet_service.cc:1468] Processing CreateTablet for tablet 78febd0e466543a2a51dab418bb9e9c6 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:22:44.256703 4409 tablet_service.cc:1468] Processing CreateTablet for tablet 78febd0e466543a2a51dab418bb9e9c6 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:22:44.258711 4338 tablet_service.cc:1468] Processing CreateTablet for tablet 691922bedd744294a3219a5605970305 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:22:44.260627 4338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691922bedd744294a3219a5605970305. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.262627 4335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78febd0e466543a2a51dab418bb9e9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.268571 4411 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745e6d53a45442c91d58026e7c5c0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.269290 4410 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b922492bc53645bf90e75036b5aa3a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.276664 4409 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78febd0e466543a2a51dab418bb9e9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.319994 4263 tablet_service.cc:1468] Processing CreateTablet for tablet 691922bedd744294a3219a5605970305 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:22:44.321434 4263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691922bedd744294a3219a5605970305. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.327559 4262 tablet_service.cc:1468] Processing CreateTablet for tablet 1745e6d53a45442c91d58026e7c5c0c1 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:22:44.330386 4260 tablet_service.cc:1468] Processing CreateTablet for tablet 78febd0e466543a2a51dab418bb9e9c6 (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:22:44.328567 4261 tablet_service.cc:1468] Processing CreateTablet for tablet b922492bc53645bf90e75036b5aa3a8a (DEFAULT_TABLE table=test-workload [id=d2ae6f32b5c34a0b961ab2764362db50]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:22:44.331287 4262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745e6d53a45442c91d58026e7c5c0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.344738 4260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78febd0e466543a2a51dab418bb9e9c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.345820 4261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b922492bc53645bf90e75036b5aa3a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:44.365880 4467 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:44.367245 4468 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:44.375362 4468 tablet_bootstrap.cc:654] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.382396 4467 tablet_bootstrap.cc:654] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.390086 4469 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:44.396997 4469 tablet_bootstrap.cc:654] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.400103 4468 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: No bootstrap required, opened a new log
I20250905 02:22:44.400645 4468 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.024s
I20250905 02:22:44.403151 4467 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: No bootstrap required, opened a new log
I20250905 02:22:44.403652 4467 ts_tablet_manager.cc:1397] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.000s
I20250905 02:22:44.404640 4469 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: No bootstrap required, opened a new log
I20250905 02:22:44.404560 4468 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.405162 4469 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20250905 02:22:44.405220 4468 raft_consensus.cc:383] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.405673 4468 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.406440 4468 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.406492 4467 raft_consensus.cc:357] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.407404 4467 raft_consensus.cc:383] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.407699 4467 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.408061 4469 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.408812 4469 raft_consensus.cc:383] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.408489 4467 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.409122 4469 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.410117 4469 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.415709 4468 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20250905 02:22:44.416726 4468 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:44.418154 4469 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.013s user 0.006s sys 0.004s
I20250905 02:22:44.419451 4469 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:44.422991 4468 tablet_bootstrap.cc:654] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.425643 4469 tablet_bootstrap.cc:654] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.434175 4467 ts_tablet_manager.cc:1428] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.030s user 0.005s sys 0.009s
I20250905 02:22:44.435256 4467 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:44.435562 4468 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: No bootstrap required, opened a new log
I20250905 02:22:44.436084 4468 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.005s
I20250905 02:22:44.436112 4469 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: No bootstrap required, opened a new log
I20250905 02:22:44.436856 4469 ts_tablet_manager.cc:1397] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20250905 02:22:44.439194 4468 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.440037 4468 raft_consensus.cc:383] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.439520 4469 raft_consensus.cc:357] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.440382 4468 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.440554 4469 raft_consensus.cc:383] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.440965 4469 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.441671 4467 tablet_bootstrap.cc:654] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.441304 4468 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.441715 4469 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.443749 4468 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 02:22:44.444303 4469 ts_tablet_manager.cc:1428] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250905 02:22:44.447036 4469 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:44.448683 4468 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:44.453614 4469 tablet_bootstrap.cc:654] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.455875 4468 tablet_bootstrap.cc:654] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.460454 4467 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: No bootstrap required, opened a new log
I20250905 02:22:44.460884 4469 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: No bootstrap required, opened a new log
I20250905 02:22:44.460983 4467 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.007s
I20250905 02:22:44.461635 4469 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20250905 02:22:44.464497 4469 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.465306 4469 raft_consensus.cc:383] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.465718 4469 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.465096 4467 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.466277 4467 raft_consensus.cc:383] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.466575 4467 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.466653 4469 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.467259 4467 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.468420 4468 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: No bootstrap required, opened a new log
I20250905 02:22:44.468879 4468 ts_tablet_manager.cc:1397] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20250905 02:22:44.469015 4469 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 02:22:44.470043 4469 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:44.471787 4468 raft_consensus.cc:357] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.472572 4468 raft_consensus.cc:383] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.472848 4468 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.473896 4467 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20250905 02:22:44.473564 4468 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.474896 4467 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:44.476326 4468 ts_tablet_manager.cc:1428] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250905 02:22:44.477506 4468 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:44.483837 4468 tablet_bootstrap.cc:654] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.476441 4469 tablet_bootstrap.cc:654] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.483837 4467 tablet_bootstrap.cc:654] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.489980 4469 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: No bootstrap required, opened a new log
I20250905 02:22:44.490572 4469 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20250905 02:22:44.493364 4469 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.493882 4469 raft_consensus.cc:383] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.494100 4469 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.494661 4469 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.496815 4469 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250905 02:22:44.498278 4467 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: No bootstrap required, opened a new log
I20250905 02:22:44.498808 4467 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.004s
I20250905 02:22:44.499752 4468 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: No bootstrap required, opened a new log
I20250905 02:22:44.500164 4468 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.010s
I20250905 02:22:44.502442 4468 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.502516 4467 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.503198 4468 raft_consensus.cc:383] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.503455 4467 raft_consensus.cc:383] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.503561 4468 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.503775 4467 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.504456 4468 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.504704 4467 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.506446 4468 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 02:22:44.508097 4467 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250905 02:22:44.509292 4467 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:44.515436 4467 tablet_bootstrap.cc:654] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:44.521304 4467 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: No bootstrap required, opened a new log
I20250905 02:22:44.521752 4467 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.005s
I20250905 02:22:44.523665 4467 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.524176 4467 raft_consensus.cc:383] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:44.524420 4467 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:44.524963 4467 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.526978 4467 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 02:22:44.607136 4473 raft_consensus.cc:491] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:44.607648 4473 raft_consensus.cc:513] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.609616 4473 leader_election.cc:290] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:44.619326 4475 raft_consensus.cc:491] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:44.619856 4475 raft_consensus.cc:513] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.623466 4273 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54835050b80544d880704fa9291f1905" is_pre_election: true
I20250905 02:22:44.623457 4348 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:44.624675 4348 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 0.
I20250905 02:22:44.625801 4475 leader_election.cc:290] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:44.626657 4386 leader_election.cc:304] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:44.627803 4473 raft_consensus.cc:2802] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:44.628242 4473 raft_consensus.cc:491] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:44.628621 4473 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.628700 4273 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 0.
I20250905 02:22:44.635396 4348 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:44.635998 4348 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 0.
I20250905 02:22:44.636309 4473 raft_consensus.cc:513] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.637164 4237 leader_election.cc:304] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:44.637964 4475 raft_consensus.cc:2802] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:44.638258 4475 raft_consensus.cc:491] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:44.638583 4475 raft_consensus.cc:3058] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.638716 4422 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b" is_pre_election: true
I20250905 02:22:44.639091 4348 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:44.639483 4422 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 0.
I20250905 02:22:44.639711 4273 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54835050b80544d880704fa9291f1905"
I20250905 02:22:44.639815 4348 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.640256 4273 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.641362 4473 leader_election.cc:290] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:44.645653 4273 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 1.
I20250905 02:22:44.646078 4475 raft_consensus.cc:513] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.646680 4387 leader_election.cc:304] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 54835050b80544d880704fa9291f1905; no voters:
I20250905 02:22:44.647389 4473 raft_consensus.cc:2802] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:44.647410 4348 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 1.
I20250905 02:22:44.648167 4475 leader_election.cc:290] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:44.649195 4347 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:44.649420 4422 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:44.649866 4347 raft_consensus.cc:3058] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.649930 4422 raft_consensus.cc:3058] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.655861 4475 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:44.656355 4475 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.657166 4347 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:44.658300 4237 leader_election.cc:304] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:44.659200 4422 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:44.659855 4421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b" is_pre_election: true
I20250905 02:22:44.660955 4421 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 0.
I20250905 02:22:44.662367 4490 raft_consensus.cc:2802] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:44.663137 4347 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:44.663764 4347 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 0.
I20250905 02:22:44.662418 4236 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 54835050b80544d880704fa9291f1905; no voters:
I20250905 02:22:44.664395 4473 raft_consensus.cc:695] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 LEADER]: Becoming Leader. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Running, Role: LEADER
I20250905 02:22:44.665306 4473 consensus_queue.cc:237] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:44.666131 4491 raft_consensus.cc:2802] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:44.665661 4490 raft_consensus.cc:695] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 LEADER]: Becoming Leader. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:44.666662 4491 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:44.667124 4491 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.667196 4475 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:44.667289 4490 consensus_queue.cc:237] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.677789 4491 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.681021 4491 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:44.680939 4347 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:44.681098 4421 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:44.681736 4347 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.681821 4421 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.683342 4170 catalog_manager.cc:5582] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b reported cstate change: term changed from 0 to 1, leader changed from <none> to 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131). New cstate: current_term: 1 leader_uuid: "437e86f2e6c646c59fd56f2315f1f81b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:44.684440 4169 catalog_manager.cc:5582] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54835050b80544d880704fa9291f1905 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:44.688733 4347 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:44.689652 4237 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:44.690443 4491 raft_consensus.cc:2802] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:44.690871 4491 raft_consensus.cc:695] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 LEADER]: Becoming Leader. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:44.691171 4421 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:44.691663 4491 consensus_queue.cc:237] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.700541 4169 catalog_manager.cc:5582] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 reported cstate change: term changed from 0 to 1, leader changed from <none> to 54835050b80544d880704fa9291f1905 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } }
W20250905 02:22:44.752946 4218 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20250905 02:22:44.786798 4473 raft_consensus.cc:491] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:44.787297 4473 raft_consensus.cc:513] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.788944 4473 leader_election.cc:290] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
I20250905 02:22:44.789880 4273 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54835050b80544d880704fa9291f1905" is_pre_election: true
I20250905 02:22:44.790046 4347 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:44.790474 4273 raft_consensus.cc:2466] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 0.
I20250905 02:22:44.790592 4347 raft_consensus.cc:2466] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 0.
I20250905 02:22:44.791482 4387 leader_election.cc:304] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 54835050b80544d880704fa9291f1905; no voters:
I20250905 02:22:44.792165 4473 raft_consensus.cc:2802] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:22:44.792562 4473 raft_consensus.cc:491] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:44.792838 4473 raft_consensus.cc:3058] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.797860 4473 raft_consensus.cc:513] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.799520 4473 leader_election.cc:290] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 election: Requested vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
I20250905 02:22:44.800698 4347 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:44.800698 4273 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54835050b80544d880704fa9291f1905"
I20250905 02:22:44.801462 4347 raft_consensus.cc:3058] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.801510 4273 raft_consensus.cc:3058] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:44.809463 4347 raft_consensus.cc:2466] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 1.
I20250905 02:22:44.809459 4273 raft_consensus.cc:2466] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 1.
I20250905 02:22:44.811025 4387 leader_election.cc:304] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 54835050b80544d880704fa9291f1905; no voters:
I20250905 02:22:44.811810 4473 raft_consensus.cc:2802] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:44.812242 4473 raft_consensus.cc:695] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 LEADER]: Becoming Leader. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Running, Role: LEADER
I20250905 02:22:44.812960 4473 consensus_queue.cc:237] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:44.819964 4169 catalog_manager.cc:5582] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b reported cstate change: term changed from 0 to 1, leader changed from <none> to 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131). New cstate: current_term: 1 leader_uuid: "437e86f2e6c646c59fd56f2315f1f81b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:44.866902 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:22:44.887315 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:44.888046 31938 tablet_replica.cc:331] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: stopping tablet replica
I20250905 02:22:44.888660 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.889016 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.890790 31938 tablet_replica.cc:331] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: stopping tablet replica
I20250905 02:22:44.891243 31938 raft_consensus.cc:2241] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:44.891753 31938 pending_rounds.cc:70] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Trying to abort 1 pending ops.
I20250905 02:22:44.891911 31938 pending_rounds.cc:77] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831599384805376 op_type: NO_OP noop_request { }
I20250905 02:22:44.892156 31938 raft_consensus.cc:2887] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:44.892409 31938 raft_consensus.cc:2270] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.894480 31938 tablet_replica.cc:331] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: stopping tablet replica
I20250905 02:22:44.894907 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:44.895385 31938 pending_rounds.cc:70] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Trying to abort 1 pending ops.
I20250905 02:22:44.895529 31938 pending_rounds.cc:77] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831599290105856 op_type: NO_OP noop_request { }
I20250905 02:22:44.895733 31938 raft_consensus.cc:2887] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:44.895932 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.898034 31938 tablet_replica.cc:331] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: stopping tablet replica
I20250905 02:22:44.898458 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.898854 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.912159 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
I20250905 02:22:44.922655 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:22:44.939916 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:44.940622 31938 tablet_replica.cc:331] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: stopping tablet replica
I20250905 02:22:44.941388 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.941766 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.943496 31938 tablet_replica.cc:331] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: stopping tablet replica
I20250905 02:22:44.943960 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.944280 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.945948 31938 tablet_replica.cc:331] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: stopping tablet replica
I20250905 02:22:44.946405 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.946717 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.948416 31938 tablet_replica.cc:331] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: stopping tablet replica
I20250905 02:22:44.948866 31938 raft_consensus.cc:2241] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.949182 31938 raft_consensus.cc:2270] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.969162 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:22:44.979530 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:22:44.994550 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:44.995384 31938 tablet_replica.cc:331] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: stopping tablet replica
I20250905 02:22:44.995977 31938 raft_consensus.cc:2241] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:44.996352 31938 raft_consensus.cc:2270] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:44.997997 31938 tablet_replica.cc:331] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: stopping tablet replica
I20250905 02:22:44.998474 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:44.999191 31938 pending_rounds.cc:70] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Trying to abort 1 pending ops.
I20250905 02:22:44.999348 31938 pending_rounds.cc:77] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831599880462336 op_type: NO_OP noop_request { }
I20250905 02:22:44.999577 31938 raft_consensus.cc:2887] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:44.999787 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:45.001662 31938 tablet_replica.cc:331] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: stopping tablet replica
I20250905 02:22:45.002105 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:45.002408 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:45.003907 31938 tablet_replica.cc:331] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: stopping tablet replica
I20250905 02:22:45.004316 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:45.004859 31938 pending_rounds.cc:70] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Trying to abort 1 pending ops.
I20250905 02:22:45.005005 31938 pending_rounds.cc:77] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831599277604864 op_type: NO_OP noop_request { }
I20250905 02:22:45.005229 31938 raft_consensus.cc:2887] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:45.005436 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:45.027431 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:22:45.038761 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:45.043982 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:45.044981 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:45.045820 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:45.046425 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:45.047405 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:45.047669 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:45.047875 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038965047864 us; error 0 us; skew 500 ppm
I20250905 02:22:45.048450 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:45.050736 31938 webserver.cc:480] Webserver started at http://127.31.48.132:43729/ using document root <none> and password file <none>
I20250905 02:22:45.051235 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:45.051479 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:45.051766 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:45.052896 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "c41710f0cfb64170a41bc535f47e8fa6"
format_stamp: "Formatted at 2025-09-05 02:22:45 on dist-test-slave-q32q"
I20250905 02:22:45.057144 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:45.060613 4505 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:45.061705 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:22:45.062103 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "c41710f0cfb64170a41bc535f47e8fa6"
format_stamp: "Formatted at 2025-09-05 02:22:45 on dist-test-slave-q32q"
I20250905 02:22:45.062575 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:45.094444 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:45.095626 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:45.097075 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:45.099272 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:45.099582 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:45.099881 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:45.100126 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:45.135455 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:33179
I20250905 02:22:45.135546 4567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:33179 every 8 connection(s)
I20250905 02:22:45.149818 4568 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:45.150290 4568 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:45.151032 4568 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:45.152908 4169 ts_manager.cc:194] Registered new tserver with Master: c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132:33179)
I20250905 02:22:45.154302 4169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59770
W20250905 02:22:45.764003 4218 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:45.767282 4157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20250905 02:22:45.770412 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:45.770843 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:45.775131 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:45.775519 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:45.779881 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:45.780261 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
I20250905 02:22:46.156754 4568 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
W20250905 02:22:46.796186 4218 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:46.806703 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:46.806950 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:46.810925 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:46.811161 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:46.815073 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:46.815284 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:47.826633 4218 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:47.832516 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:47.832768 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:47.836544 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:47.836772 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:47.840626 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:47.840883 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:48.851852 4218 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:48.857160 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:48.857415 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.131:46329: connect: Connection refused (error 111)
W20250905 02:22:48.861213 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:48.861474 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:48.865482 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:48.865715 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:49.868398 4218 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6c7070b76dbc426784bf84a6040700d1 not available for placement
W20250905 02:22:50.954057 4218 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 6c7070b76dbc426784bf84a6040700d1 not available for placement
I20250905 02:22:51.329927 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:51.335369 4584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:51.336148 4585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:51.339706 4587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:51.341423 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:51.342430 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:51.342610 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:51.342784 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038971342766 us; error 0 us; skew 500 ppm
I20250905 02:22:51.343390 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:51.345749 31938 webserver.cc:480] Webserver started at http://127.31.48.129:36261/ using document root <none> and password file <none>
I20250905 02:22:51.346212 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:51.346364 31938 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 02:22:51.349951 31938 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20250905 02:22:51.352965 4592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:51.353719 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:22:51.354099 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "54835050b80544d880704fa9291f1905"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:51.354391 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:51.372933 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:51.373996 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:51.375522 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:51.379223 4599 ts_tablet_manager.cc:542] Loading tablet metadata (0/4 complete)
I20250905 02:22:51.397595 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250905 02:22:51.397864 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.020s user 0.000s sys 0.001s
I20250905 02:22:51.398105 31938 ts_tablet_manager.cc:594] Registering tablets (0/4 complete)
I20250905 02:22:51.402285 4599 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:51.411468 31938 ts_tablet_manager.cc:610] Registered 4 tablets
I20250905 02:22:51.411712 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.014s user 0.011s sys 0.000s
I20250905 02:22:51.412444 4599 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.413144 4599 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Bootstrap complete.
I20250905 02:22:51.413687 4599 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20250905 02:22:51.415450 4599 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.415930 4599 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.416513 4599 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.419206 4599 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250905 02:22:51.419852 4599 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:51.430742 4599 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:22:51.431690 4599 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Bootstrap complete.
I20250905 02:22:51.432207 4599 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20250905 02:22:51.433903 4599 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.434522 4599 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.435087 4599 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.436764 4599 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:51.437418 4599 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:51.448762 4599 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.449585 4599 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Bootstrap complete.
I20250905 02:22:51.450160 4599 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250905 02:22:51.451978 4599 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:51.452555 4599 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.453190 4599 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:51.454843 4599 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250905 02:22:51.455502 4599 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Bootstrap starting.
I20250905 02:22:51.493463 4599 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:22:51.494688 4599 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Bootstrap complete.
I20250905 02:22:51.495424 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:33149
I20250905 02:22:51.495652 4599 ts_tablet_manager.cc:1397] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.005s
I20250905 02:22:51.495573 4661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:33149 every 8 connection(s)
I20250905 02:22:51.499329 4599 raft_consensus.cc:357] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.501561 4599 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Initialized, Role: FOLLOWER
W20250905 02:22:51.502596 4663 tablet.cc:2378] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 02:22:51.503275 4599 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.504273 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:22:51.507117 4599 ts_tablet_manager.cc:1428] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905: Time spent starting tablet: real 0.011s user 0.002s sys 0.007s
W20250905 02:22:51.518332 4665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:51.522830 4667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:51.524418 4669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:51.526674 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:51.527505 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:51.527699 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:51.527878 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038971527862 us; error 0 us; skew 500 ppm
I20250905 02:22:51.528451 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:51.531488 31938 webserver.cc:480] Webserver started at http://127.31.48.130:44057/ using document root <none> and password file <none>
I20250905 02:22:51.532030 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:51.532264 31938 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 02:22:51.532550 4662 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:51.532904 4662 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:51.533654 4662 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:51.536653 31938 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20250905 02:22:51.538329 4169 ts_manager.cc:194] Re-registered known tserver with Master: 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:51.541476 4674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:51.542430 31938 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250905 02:22:51.542701 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "6c7070b76dbc426784bf84a6040700d1"
format_stamp: "Formatted at 2025-09-05 02:22:42 on dist-test-slave-q32q"
I20250905 02:22:51.543073 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:51.548732 4169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44480
I20250905 02:22:51.552346 4662 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
I20250905 02:22:51.558657 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:51.560009 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:51.561662 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:51.584874 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250905 02:22:51.585131 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.000s sys 0.002s
I20250905 02:22:51.590010 4681 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:51.603863 4681 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.604687 4681 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Bootstrap complete.
I20250905 02:22:51.605329 4681 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20250905 02:22:51.606005 31938 ts_tablet_manager.cc:610] Registered 4 tablets
I20250905 02:22:51.606289 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.021s user 0.019s sys 0.001s
I20250905 02:22:51.607934 4681 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.608731 4681 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.609503 4681 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.612493 4681 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20250905 02:22:51.613394 4681 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:51.629539 4681 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.630205 4681 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap complete.
I20250905 02:22:51.630753 4681 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20250905 02:22:51.632928 4681 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.633476 4681 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.634057 4681 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.635669 4681 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 02:22:51.636301 4681 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:51.648736 4681 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.649945 4681 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap complete.
I20250905 02:22:51.650684 4681 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20250905 02:22:51.653613 4681 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:51.654462 4681 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.655380 4681 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:51.658313 4681 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250905 02:22:51.659219 4681 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap starting.
I20250905 02:22:51.662792 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:36981
I20250905 02:22:51.662952 4741 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:36981 every 8 connection(s)
I20250905 02:22:51.668402 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:22:51.676586 4681 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20250905 02:22:51.678261 4747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:51.679888 4681 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Bootstrap complete.
W20250905 02:22:51.680780 4746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:51.680939 4681 ts_tablet_manager.cc:1397] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Time spent bootstrapping tablet: real 0.022s user 0.015s sys 0.004s
I20250905 02:22:51.684139 4742 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:51.684698 4742 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:51.684551 4681 raft_consensus.cc:357] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:51.685194 4749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:51.685460 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:51.685321 4681 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.685895 4742 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:51.686352 4681 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.687006 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:51.687316 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:51.687520 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038971687504 us; error 0 us; skew 500 ppm
I20250905 02:22:51.688243 4681 ts_tablet_manager.cc:1428] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20250905 02:22:51.688324 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:51.690963 4169 ts_manager.cc:194] Re-registered known tserver with Master: 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
I20250905 02:22:51.691119 31938 webserver.cc:480] Webserver started at http://127.31.48.131:34057/ using document root <none> and password file <none>
I20250905 02:22:51.691701 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:51.691860 31938 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 02:22:51.695657 31938 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20250905 02:22:51.696079 4169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44486
I20250905 02:22:51.699383 4754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:51.699447 4742 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
I20250905 02:22:51.700312 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:22:51.700690 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "437e86f2e6c646c59fd56f2315f1f81b"
format_stamp: "Formatted at 2025-09-05 02:22:43 on dist-test-slave-q32q"
I20250905 02:22:51.701042 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestHandlingFailedTservers.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:51.715083 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:51.716293 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:51.717701 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:51.740914 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (4 total tablets, 4 live tablets)
I20250905 02:22:51.741180 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.000s sys 0.002s
I20250905 02:22:51.745963 4761 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:51.761202 4761 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:22:51.762130 4761 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap complete.
I20250905 02:22:51.762562 31938 ts_tablet_manager.cc:610] Registered 4 tablets
I20250905 02:22:51.762748 4761 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20250905 02:22:51.762868 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.021s user 0.018s sys 0.001s
I20250905 02:22:51.765667 4761 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.766598 4761 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.767378 4761 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.770257 4761 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20250905 02:22:51.771131 4761 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:51.787598 4761 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.788324 4761 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap complete.
I20250905 02:22:51.788897 4761 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20250905 02:22:51.790813 4761 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.791363 4761 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.791955 4761 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.793632 4761 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 02:22:51.794261 4761 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:51.807152 4761 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:22:51.808271 4761 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap complete.
I20250905 02:22:51.809024 4761 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20250905 02:22:51.812017 4761 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:51.813112 4761 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.813962 4761 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:51.816426 4761 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 02:22:51.817385 4761 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap starting.
I20250905 02:22:51.817642 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:46329
I20250905 02:22:51.817835 4821 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:46329 every 8 connection(s)
I20250905 02:22:51.830518 4761 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:51.831626 4761 tablet_bootstrap.cc:492] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Bootstrap complete.
I20250905 02:22:51.832298 4761 ts_tablet_manager.cc:1397] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.001s
I20250905 02:22:51.835143 4761 raft_consensus.cc:357] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.836134 4761 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Initialized, Role: FOLLOWER
I20250905 02:22:51.837152 4761 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:51.838765 4822 heartbeater.cc:344] Connected to a master server at 127.31.48.190:34765
I20250905 02:22:51.839329 4822 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:51.839454 4761 ts_tablet_manager.cc:1428] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 02:22:51.840525 4822 heartbeater.cc:507] Master 127.31.48.190:34765 requested a full tablet report, sending...
I20250905 02:22:51.843923 4169 ts_manager.cc:194] Re-registered known tserver with Master: 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:51.848886 4169 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44492
I20250905 02:22:51.851145 4822 heartbeater.cc:499] Master 127.31.48.190:34765 was elected leader, sending a full tablet report...
W20250905 02:22:51.943022 4564 debug-util.cc:398] Leaking SignalData structure 0x7b080008e680 after lost signal to thread 31941
W20250905 02:22:51.944950 4564 debug-util.cc:398] Leaking SignalData structure 0x7b08001b4ca0 after lost signal to thread 4567
W20250905 02:22:51.978817 4218 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 54835050b80544d880704fa9291f1905 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } }
W20250905 02:22:51.983163 4218 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 1745e6d53a45442c91d58026e7c5c0c1, TS 437e86f2e6c646c59fd56f2315f1f81b -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:51.983659 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 1745e6d53a45442c91d58026e7c5c0c1, TS 437e86f2e6c646c59fd56f2315f1f81b -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:51.996869 4218 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b922492bc53645bf90e75036b5aa3a8a, TS 54835050b80544d880704fa9291f1905 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:51.997489 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b922492bc53645bf90e75036b5aa3a8a, TS 54835050b80544d880704fa9291f1905 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:52.001556 4218 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 691922bedd744294a3219a5605970305, TS 6c7070b76dbc426784bf84a6040700d1 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
W20250905 02:22:52.002132 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 691922bedd744294a3219a5605970305, TS 6c7070b76dbc426784bf84a6040700d1 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:52.730624 4829 raft_consensus.cc:491] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:52.731164 4829 raft_consensus.cc:513] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.733181 4829 leader_election.cc:290] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:52.751065 4829 raft_consensus.cc:491] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:52.751605 4829 raft_consensus.cc:513] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.753659 4829 leader_election.cc:290] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:52.759344 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:52.759822 4715 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:52.760141 4716 raft_consensus.cc:2466] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:52.760507 4715 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:52.762332 4796 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b" is_pre_election: true
I20250905 02:22:52.762574 4594 leader_election.cc:304] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:52.762760 4795 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b" is_pre_election: true
I20250905 02:22:52.763286 4796 raft_consensus.cc:2408] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54835050b80544d880704fa9291f1905 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250905 02:22:52.763351 4829 raft_consensus.cc:2802] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:52.763530 4795 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:52.763875 4829 raft_consensus.cc:491] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:52.764267 4829 raft_consensus.cc:3058] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.764935 4594 leader_election.cc:304] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:52.767009 4834 raft_consensus.cc:2802] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:52.767421 4834 raft_consensus.cc:491] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:52.767835 4834 raft_consensus.cc:3058] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.771512 4829 raft_consensus.cc:513] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.773155 4829 leader_election.cc:290] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:52.773893 4715 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:52.774262 4795 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b922492bc53645bf90e75036b5aa3a8a" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:52.774586 4715 raft_consensus.cc:3058] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.774928 4795 raft_consensus.cc:3058] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.776485 4834 raft_consensus.cc:513] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.779124 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:52.779680 4716 raft_consensus.cc:3058] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.782112 4715 raft_consensus.cc:2466] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 2.
I20250905 02:22:52.783119 4834 leader_election.cc:290] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:52.784276 4796 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1745e6d53a45442c91d58026e7c5c0c1" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:52.785154 4796 raft_consensus.cc:3058] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.782915 4829 raft_consensus.cc:491] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:52.789263 4716 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 2.
I20250905 02:22:52.789165 4829 raft_consensus.cc:513] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:52.789425 4594 leader_election.cc:304] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:52.791246 4594 leader_election.cc:304] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:52.792075 4834 raft_consensus.cc:2802] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:52.792590 4796 raft_consensus.cc:2466] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 2.
I20250905 02:22:52.788374 4795 raft_consensus.cc:2408] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54835050b80544d880704fa9291f1905 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250905 02:22:52.794977 4835 raft_consensus.cc:2802] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:52.796007 4829 leader_election.cc:290] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:52.796300 4835 raft_consensus.cc:695] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Becoming Leader. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:52.796810 4834 raft_consensus.cc:695] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Becoming Leader. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:52.797303 4835 consensus_queue.cc:237] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.797806 4796 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b" is_pre_election: true
I20250905 02:22:52.797873 4834 consensus_queue.cc:237] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.798887 4796 raft_consensus.cc:2408] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 54835050b80544d880704fa9291f1905 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250905 02:22:52.795184 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:52.800686 4716 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 1.
I20250905 02:22:52.802209 4594 leader_election.cc:304] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters: 437e86f2e6c646c59fd56f2315f1f81b
I20250905 02:22:52.803973 4836 raft_consensus.cc:2802] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:52.804527 4836 raft_consensus.cc:491] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:52.805039 4836 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.814630 4836 raft_consensus.cc:513] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:52.817777 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:52.818368 4716 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.818938 4795 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:52.819581 4795 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:52.819324 4169 catalog_manager.cc:5582] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 reported cstate change: term changed from 1 to 2, leader changed from 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131) to 54835050b80544d880704fa9291f1905 (127.31.48.129). New cstate: current_term: 2 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:52.821292 4169 catalog_manager.cc:5582] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:52.825803 4716 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 2.
I20250905 02:22:52.827409 4795 raft_consensus.cc:2408] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 54835050b80544d880704fa9291f1905 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250905 02:22:52.827486 4594 leader_election.cc:304] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:52.828929 4834 raft_consensus.cc:2802] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:22:52.829619 4834 raft_consensus.cc:695] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Becoming Leader. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:52.830554 4836 leader_election.cc:290] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 2 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:52.830771 4834 consensus_queue.cc:237] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:52.855829 4169 catalog_manager.cc:5582] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 reported cstate change: term changed from 1 to 2, leader changed from 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131) to 54835050b80544d880704fa9291f1905 (127.31.48.129). New cstate: current_term: 2 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:52.979820 4844 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:52.980315 4844 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:52.982592 4844 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
I20250905 02:22:53.007205 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:53.007966 4716 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 1.
I20250905 02:22:53.009531 4634 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54835050b80544d880704fa9291f1905" is_pre_election: true
I20250905 02:22:53.009662 4756 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:53.010450 4634 raft_consensus.cc:2408] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 437e86f2e6c646c59fd56f2315f1f81b for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250905 02:22:53.010633 4844 raft_consensus.cc:2802] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:22:53.010974 4844 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:53.011320 4844 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 1 FOLLOWER]: Advancing to term 2
W20250905 02:22:53.017655 4218 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 54835050b80544d880704fa9291f1905 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } }
I20250905 02:22:53.017956 4844 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.020641 4844 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 2 election: Requested vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
I20250905 02:22:53.021317 4633 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54835050b80544d880704fa9291f1905"
I20250905 02:22:53.021560 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "437e86f2e6c646c59fd56f2315f1f81b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:53.023782 4633 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:22:53.023787 4716 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 1 FOLLOWER]: Advancing to term 2
W20250905 02:22:53.023931 4218 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 78febd0e466543a2a51dab418bb9e9c6, TS 6c7070b76dbc426784bf84a6040700d1 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:53.024798 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 78febd0e466543a2a51dab418bb9e9c6, TS 6c7070b76dbc426784bf84a6040700d1 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
W20250905 02:22:53.030028 4218 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 691922bedd744294a3219a5605970305, TS 437e86f2e6c646c59fd56f2315f1f81b -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
W20250905 02:22:53.030457 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 691922bedd744294a3219a5605970305, TS 437e86f2e6c646c59fd56f2315f1f81b -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } }
I20250905 02:22:53.031672 4716 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 437e86f2e6c646c59fd56f2315f1f81b in term 2.
I20250905 02:22:53.032006 4633 raft_consensus.cc:2408] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 437e86f2e6c646c59fd56f2315f1f81b for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20250905 02:22:53.032948 4756 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:53.033826 4844 raft_consensus.cc:2802] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader election won for term 2
W20250905 02:22:53.034966 4218 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet b922492bc53645bf90e75036b5aa3a8a, TS 54835050b80544d880704fa9291f1905 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.035200 4844 raft_consensus.cc:695] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 LEADER]: Becoming Leader. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Running, Role: LEADER
W20250905 02:22:53.035423 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet b922492bc53645bf90e75036b5aa3a8a, TS 54835050b80544d880704fa9291f1905 -> TS c41710f0cfb64170a41bc535f47e8fa6 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.036103 4844 consensus_queue.cc:237] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.044426 4169 catalog_manager.cc:5582] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b reported cstate change: term changed from 1 to 2, leader changed from 54835050b80544d880704fa9291f1905 (127.31.48.129) to 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131). New cstate: current_term: 2 leader_uuid: "437e86f2e6c646c59fd56f2315f1f81b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: HEALTHY } } }
I20250905 02:22:53.132761 4834 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:53.133412 4834 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.135777 4834 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:53.136495 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1" is_pre_election: true
I20250905 02:22:53.136801 4795 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b" is_pre_election: true
I20250905 02:22:53.137030 4716 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 2.
I20250905 02:22:53.137943 4594 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:53.138754 4834 raft_consensus.cc:2802] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 02:22:53.139142 4834 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:22:53.139504 4834 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:53.147095 4834 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.149108 4834 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 3 election: Requested vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329)
I20250905 02:22:53.149968 4716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
I20250905 02:22:53.150244 4795 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78febd0e466543a2a51dab418bb9e9c6" candidate_uuid: "54835050b80544d880704fa9291f1905" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:53.150579 4716 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:53.157996 4716 raft_consensus.cc:2466] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 54835050b80544d880704fa9291f1905 in term 3.
I20250905 02:22:53.159097 4594 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1; no voters: 437e86f2e6c646c59fd56f2315f1f81b
I20250905 02:22:53.159988 4834 raft_consensus.cc:2802] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:53.160426 4834 raft_consensus.cc:695] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 3 LEADER]: Becoming Leader. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:53.161163 4834 consensus_queue.cc:237] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.168081 4169 catalog_manager.cc:5582] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 reported cstate change: term changed from 2 to 3, leader changed from 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131) to 54835050b80544d880704fa9291f1905 (127.31.48.129). New cstate: current_term: 3 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:53.234025 4795 raft_consensus.cc:1273] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:53.235226 4834 consensus_queue.cc:1035] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.252260 4716 raft_consensus.cc:1273] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:53.255193 4715 raft_consensus.cc:1273] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250905 02:22:53.257323 4834 consensus_queue.cc:1035] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.272514 4836 consensus_queue.cc:1035] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:22:53.295710 4795 raft_consensus.cc:1273] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250905 02:22:53.296146 4795 pending_rounds.cc:85] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Aborting all ops after (but not including) 0
I20250905 02:22:53.296422 4795 pending_rounds.cc:107] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250905 02:22:53.296710 4795 raft_consensus.cc:2887] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 02:22:53.300408 4835 consensus_queue.cc:1035] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.362455 4795 raft_consensus.cc:1273] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250905 02:22:53.362804 4795 pending_rounds.cc:85] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Aborting all ops after (but not including) 0
I20250905 02:22:53.363008 4795 pending_rounds.cc:107] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250905 02:22:53.363205 4795 raft_consensus.cc:2887] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 02:22:53.364482 4835 consensus_queue.cc:1035] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.373922 4715 raft_consensus.cc:1273] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20250905 02:22:53.375582 4834 consensus_queue.cc:1035] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.475783 4634 raft_consensus.cc:1238] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 3 LEADER]: Rejecting Update request from peer 437e86f2e6c646c59fd56f2315f1f81b for earlier term 2. Current term is 3. Ops: []
I20250905 02:22:53.477378 4844 consensus_queue.cc:1046] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.477985 4844 consensus_queue.cc:1230] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [LEADER]: Peer 54835050b80544d880704fa9291f1905 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250905 02:22:53.478328 4848 raft_consensus.cc:3053] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 LEADER]: Stepping down as leader of term 2
I20250905 02:22:53.478722 4848 raft_consensus.cc:738] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 437e86f2e6c646c59fd56f2315f1f81b, State: Running, Role: LEADER
I20250905 02:22:53.479628 4848 consensus_queue.cc:260] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:53.481016 4848 raft_consensus.cc:3058] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:53.697669 4795 raft_consensus.cc:1273] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 02:22:53.698874 4834 consensus_queue.cc:1035] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:22:53.699486 4834 consensus_queue.cc:1230] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [LEADER]: Peer 437e86f2e6c646c59fd56f2315f1f81b log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20250905 02:22:53.702018 4795 pending_rounds.cc:85] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Aborting all ops after (but not including) 0
I20250905 02:22:53.702265 4795 pending_rounds.cc:107] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b: Aborting uncommitted NO_OP operation due to leader change: 2.1
I20250905 02:22:53.702491 4795 raft_consensus.cc:2887] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 02:22:53.711607 4715 raft_consensus.cc:1273] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250905 02:22:53.713390 4860 consensus_queue.cc:1035] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:22:54.122186 4634 consensus_queue.cc:237] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.133266 4795 raft_consensus.cc:1273] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:54.134217 4715 raft_consensus.cc:1273] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:54.135780 4834 consensus_queue.cc:1035] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:54.138578 4834 consensus_queue.cc:1035] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:54.203079 4867 raft_consensus.cc:2953] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.210134 4715 raft_consensus.cc:2953] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
W20250905 02:22:54.215996 4593 consensus_peers.cc:489] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 -> Peer c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132:33179): Couldn't send request to peer c41710f0cfb64170a41bc535f47e8fa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b922492bc53645bf90e75036b5aa3a8a. This is attempt 1: this message will repeat every 5th retry.
W20250905 02:22:54.217226 4593 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.31.48.132:33179, user_credentials={real_user=slave}} blocked reactor thread for 69622.5us
I20250905 02:22:54.221007 4795 raft_consensus.cc:2953] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.221524 4169 catalog_manager.cc:5582] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 reported cstate change: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:54.234436 4634 consensus_queue.cc:237] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.242296 4715 raft_consensus.cc:1273] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:22:54.243964 4835 consensus_queue.cc:1035] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250905 02:22:54.247326 4593 consensus_peers.cc:489] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 -> Peer c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132:33179): Couldn't send request to peer c41710f0cfb64170a41bc535f47e8fa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1745e6d53a45442c91d58026e7c5c0c1. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:54.248297 4795 raft_consensus.cc:1273] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:22:54.250320 4835 consensus_queue.cc:1035] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:22:54.251519 4860 raft_consensus.cc:2953] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.253392 4715 raft_consensus.cc:2953] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.268715 4634 consensus_queue.cc:237] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.268544 4795 raft_consensus.cc:2953] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.276998 4796 raft_consensus.cc:1273] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250905 02:22:54.278699 4593 consensus_peers.cc:489] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 -> Peer c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132:33179): Couldn't send request to peer c41710f0cfb64170a41bc535f47e8fa6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 691922bedd744294a3219a5605970305. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:22:54.279171 4860 consensus_queue.cc:1035] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:54.290534 4170 catalog_manager.cc:5582] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b reported cstate change: config changed from index -1 to 3, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.292650 4867 raft_consensus.cc:2953] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.294112 4795 raft_consensus.cc:2953] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.297070 4715 raft_consensus.cc:1273] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Refusing update from remote peer 54835050b80544d880704fa9291f1905: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:22:54.298840 4860 consensus_queue.cc:1035] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:22:54.304443 4715 raft_consensus.cc:2953] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.315092 4170 catalog_manager.cc:5582] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 reported cstate change: config changed from index -1 to 2, NON_VOTER c41710f0cfb64170a41bc535f47e8fa6 (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "54835050b80544d880704fa9291f1905" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } } }
I20250905 02:22:54.327090 4634 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "691922bedd744294a3219a5605970305"
dest_uuid: "54835050b80544d880704fa9291f1905"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59278
I20250905 02:22:54.327682 4634 raft_consensus.cc:604] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Received request to transfer leadership
I20250905 02:22:54.342845 4634 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "691922bedd744294a3219a5605970305"
dest_uuid: "54835050b80544d880704fa9291f1905"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59278
I20250905 02:22:54.343351 4634 raft_consensus.cc:604] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Received request to transfer leadership
W20250905 02:22:54.344246 4216 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: leadership transfer for 691922bedd744294a3219a5605970305 already in progress [suppressed 11 similar messages]
I20250905 02:22:54.347030 4634 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "691922bedd744294a3219a5605970305"
dest_uuid: "54835050b80544d880704fa9291f1905"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59278
I20250905 02:22:54.347479 4634 raft_consensus.cc:604] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Received request to transfer leadership
W20250905 02:22:54.348480 4218 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: leadership transfer for 691922bedd744294a3219a5605970305 already in progress
W20250905 02:22:54.348697 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for 691922bedd744294a3219a5605970305 already in progress
I20250905 02:22:54.653033 4875 ts_tablet_manager.cc:927] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: Initiating tablet copy from peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:54.655073 4875 tablet_copy_client.cc:323] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.129:33149
I20250905 02:22:54.685065 4645 tablet_copy_service.cc:140] P 54835050b80544d880704fa9291f1905: Received BeginTabletCopySession request for tablet 1745e6d53a45442c91d58026e7c5c0c1 from peer c41710f0cfb64170a41bc535f47e8fa6 ({username='slave'} at 127.0.0.1:59284)
I20250905 02:22:54.685592 4645 tablet_copy_service.cc:161] P 54835050b80544d880704fa9291f1905: Beginning new tablet copy session on tablet 1745e6d53a45442c91d58026e7c5c0c1 from peer c41710f0cfb64170a41bc535f47e8fa6 at {username='slave'} at 127.0.0.1:59284: session id = c41710f0cfb64170a41bc535f47e8fa6-1745e6d53a45442c91d58026e7c5c0c1
I20250905 02:22:54.692327 4645 tablet_copy_source_session.cc:215] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:54.695245 4875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1745e6d53a45442c91d58026e7c5c0c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:54.707953 4835 raft_consensus.cc:991] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: : Instructing follower 6c7070b76dbc426784bf84a6040700d1 to start an election
I20250905 02:22:54.708544 4835 raft_consensus.cc:1079] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Signalling peer 6c7070b76dbc426784bf84a6040700d1 to start an election
I20250905 02:22:54.710505 4715 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "691922bedd744294a3219a5605970305"
dest_uuid: "6c7070b76dbc426784bf84a6040700d1"
from {username='slave'} at 127.0.0.1:56344
I20250905 02:22:54.711092 4715 raft_consensus.cc:491] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 02:22:54.711467 4715 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:54.718595 4875 tablet_copy_client.cc:806] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:54.719336 4875 tablet_copy_client.cc:670] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:54.722281 4879 ts_tablet_manager.cc:927] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: Initiating tablet copy from peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:54.722362 4715 raft_consensus.cc:513] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.723330 4875 tablet_copy_client.cc:538] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:54.730968 4879 tablet_copy_client.cc:323] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.129:33149
I20250905 02:22:54.732730 4645 tablet_copy_service.cc:140] P 54835050b80544d880704fa9291f1905: Received BeginTabletCopySession request for tablet b922492bc53645bf90e75036b5aa3a8a from peer c41710f0cfb64170a41bc535f47e8fa6 ({username='slave'} at 127.0.0.1:59284)
I20250905 02:22:54.733273 4645 tablet_copy_service.cc:161] P 54835050b80544d880704fa9291f1905: Beginning new tablet copy session on tablet b922492bc53645bf90e75036b5aa3a8a from peer c41710f0cfb64170a41bc535f47e8fa6 at {username='slave'} at 127.0.0.1:59284: session id = c41710f0cfb64170a41bc535f47e8fa6-b922492bc53645bf90e75036b5aa3a8a
I20250905 02:22:54.735663 4875 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap starting.
I20250905 02:22:54.741016 4645 tablet_copy_source_session.cc:215] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:54.744580 4879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b922492bc53645bf90e75036b5aa3a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:54.763512 4715 leader_election.cc:290] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [CANDIDATE]: Term 3 election: Requested vote from peers 437e86f2e6c646c59fd56f2315f1f81b (127.31.48.131:46329), 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:54.772392 4879 tablet_copy_client.cc:806] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:54.773653 4879 tablet_copy_client.cc:670] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:54.774932 4875 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:54.782150 4875 tablet_bootstrap.cc:492] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap complete.
I20250905 02:22:54.779312 4879 tablet_copy_client.cc:538] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:54.787912 4795 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "6c7070b76dbc426784bf84a6040700d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "437e86f2e6c646c59fd56f2315f1f81b"
I20250905 02:22:54.788695 4795 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:54.788949 4875 ts_tablet_manager.cc:1397] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: Time spent bootstrapping tablet: real 0.054s user 0.017s sys 0.015s
I20250905 02:22:54.791141 4875 raft_consensus.cc:357] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.791647 4875 raft_consensus.cc:738] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c41710f0cfb64170a41bc535f47e8fa6, State: Initialized, Role: LEARNER
I20250905 02:22:54.792073 4875 consensus_queue.cc:260] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.793721 4885 ts_tablet_manager.cc:927] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: Initiating tablet copy from peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
I20250905 02:22:54.793874 4633 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "691922bedd744294a3219a5605970305" candidate_uuid: "6c7070b76dbc426784bf84a6040700d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "54835050b80544d880704fa9291f1905"
I20250905 02:22:54.794533 4633 raft_consensus.cc:3053] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 02:22:54.794862 4633 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 54835050b80544d880704fa9291f1905, State: Running, Role: LEADER
I20250905 02:22:54.795688 4633 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.797305 4633 raft_consensus.cc:3058] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Advancing to term 3
I20250905 02:22:54.800897 4879 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap starting.
I20250905 02:22:54.806929 4795 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c7070b76dbc426784bf84a6040700d1 in term 3.
I20250905 02:22:54.809228 4885 tablet_copy_client.cc:323] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.129:33149
I20250905 02:22:54.813923 4875 ts_tablet_manager.cc:1428] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: Time spent starting tablet: real 0.025s user 0.012s sys 0.008s
I20250905 02:22:54.815903 4675 leader_election.cc:304] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b, 6c7070b76dbc426784bf84a6040700d1; no voters:
I20250905 02:22:54.816746 4645 tablet_copy_service.cc:140] P 54835050b80544d880704fa9291f1905: Received BeginTabletCopySession request for tablet 691922bedd744294a3219a5605970305 from peer c41710f0cfb64170a41bc535f47e8fa6 ({username='slave'} at 127.0.0.1:59284)
I20250905 02:22:54.817036 4633 raft_consensus.cc:2466] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c7070b76dbc426784bf84a6040700d1 in term 3.
I20250905 02:22:54.817342 4645 tablet_copy_service.cc:161] P 54835050b80544d880704fa9291f1905: Beginning new tablet copy session on tablet 691922bedd744294a3219a5605970305 from peer c41710f0cfb64170a41bc535f47e8fa6 at {username='slave'} at 127.0.0.1:59284: session id = c41710f0cfb64170a41bc535f47e8fa6-691922bedd744294a3219a5605970305
I20250905 02:22:54.819221 4644 tablet_copy_service.cc:342] P 54835050b80544d880704fa9291f1905: Request end of tablet copy session c41710f0cfb64170a41bc535f47e8fa6-1745e6d53a45442c91d58026e7c5c0c1 received from {username='slave'} at 127.0.0.1:59284
I20250905 02:22:54.819692 4644 tablet_copy_service.cc:434] P 54835050b80544d880704fa9291f1905: ending tablet copy session c41710f0cfb64170a41bc535f47e8fa6-1745e6d53a45442c91d58026e7c5c0c1 on tablet 1745e6d53a45442c91d58026e7c5c0c1 with peer c41710f0cfb64170a41bc535f47e8fa6
I20250905 02:22:54.833101 4645 tablet_copy_source_session.cc:215] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:22:54.835028 4888 raft_consensus.cc:2802] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Leader election won for term 3
I20250905 02:22:54.838115 4885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 691922bedd744294a3219a5605970305. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:54.856581 4888 raft_consensus.cc:695] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 3 LEADER]: Becoming Leader. State: Replica: 6c7070b76dbc426784bf84a6040700d1, State: Running, Role: LEADER
I20250905 02:22:54.857517 4888 consensus_queue.cc:237] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.866274 4885 tablet_copy_client.cc:806] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Starting download of 0 data blocks...
I20250905 02:22:54.866917 4885 tablet_copy_client.cc:670] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Starting download of 1 WAL segments...
I20250905 02:22:54.870276 4879 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:54.869315 4171 catalog_manager.cc:5582] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 reported cstate change: term changed from 2 to 3, leader changed from 54835050b80544d880704fa9291f1905 (127.31.48.129) to 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130). New cstate: current_term: 3 leader_uuid: "6c7070b76dbc426784bf84a6040700d1" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 02:22:54.871444 4879 tablet_bootstrap.cc:492] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap complete.
I20250905 02:22:54.872462 4879 ts_tablet_manager.cc:1397] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: Time spent bootstrapping tablet: real 0.072s user 0.026s sys 0.015s
I20250905 02:22:54.876076 4879 raft_consensus.cc:357] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.878383 4879 raft_consensus.cc:738] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c41710f0cfb64170a41bc535f47e8fa6, State: Initialized, Role: LEARNER
I20250905 02:22:54.879173 4879 consensus_queue.cc:260] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.880481 4885 tablet_copy_client.cc:538] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:22:54.882511 4879 ts_tablet_manager.cc:1428] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: Time spent starting tablet: real 0.010s user 0.005s sys 0.002s
I20250905 02:22:54.887929 4645 tablet_copy_service.cc:342] P 54835050b80544d880704fa9291f1905: Request end of tablet copy session c41710f0cfb64170a41bc535f47e8fa6-b922492bc53645bf90e75036b5aa3a8a received from {username='slave'} at 127.0.0.1:59284
I20250905 02:22:54.888543 4645 tablet_copy_service.cc:434] P 54835050b80544d880704fa9291f1905: ending tablet copy session c41710f0cfb64170a41bc535f47e8fa6-b922492bc53645bf90e75036b5aa3a8a on tablet b922492bc53645bf90e75036b5aa3a8a with peer c41710f0cfb64170a41bc535f47e8fa6
I20250905 02:22:54.890702 4885 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap starting.
I20250905 02:22:54.922380 4885 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:22:54.923473 4885 tablet_bootstrap.cc:492] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: Bootstrap complete.
I20250905 02:22:54.924252 4885 ts_tablet_manager.cc:1397] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20250905 02:22:54.927233 4885 raft_consensus.cc:357] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.928170 4885 raft_consensus.cc:738] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: c41710f0cfb64170a41bc535f47e8fa6, State: Initialized, Role: LEARNER
I20250905 02:22:54.928937 4885 consensus_queue.cc:260] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:54.931494 4885 ts_tablet_manager.cc:1428] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 02:22:54.933275 4645 tablet_copy_service.cc:342] P 54835050b80544d880704fa9291f1905: Request end of tablet copy session c41710f0cfb64170a41bc535f47e8fa6-691922bedd744294a3219a5605970305 received from {username='slave'} at 127.0.0.1:59284
I20250905 02:22:54.933725 4645 tablet_copy_service.cc:434] P 54835050b80544d880704fa9291f1905: ending tablet copy session c41710f0cfb64170a41bc535f47e8fa6-691922bedd744294a3219a5605970305 on tablet 691922bedd744294a3219a5605970305 with peer c41710f0cfb64170a41bc535f47e8fa6
I20250905 02:22:55.150918 31938 tablet_server.cc:178] TabletServer@127.31.48.129:33149 shutting down...
W20250905 02:22:55.154721 4218 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:22:55.155082 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:22:55.160540 4218 auto_rebalancer.cc:663] Could not move replica: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:22:55.160917 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20250905 02:22:55.164692 4738 debug-util.cc:398] Leaking SignalData structure 0x7b080019fce0 after lost signal to thread 31941
W20250905 02:22:55.165606 4738 debug-util.cc:398] Leaking SignalData structure 0x7b0800228620 after lost signal to thread 4205
W20250905 02:22:55.166880 4738 debug-util.cc:398] Leaking SignalData structure 0x7b0800071980 after lost signal to thread 4567
W20250905 02:22:55.168498 4738 debug-util.cc:398] Leaking SignalData structure 0x7b080004c000 after lost signal to thread 4741
I20250905 02:22:55.173296 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:55.173940 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.174562 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:55.175016 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P 54835050b80544d880704fa9291f1905 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.177630 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.178117 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:22:55.179030 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.181371 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.181874 31938 raft_consensus.cc:2241] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 3 LEADER]: Raft consensus shutting down.
I20250905 02:22:55.182581 31938 raft_consensus.cc:2270] T 78febd0e466543a2a51dab418bb9e9c6 P 54835050b80544d880704fa9291f1905 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.184633 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.185361 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:22:55.186575 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P 54835050b80544d880704fa9291f1905 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.201145 31938 tablet_server.cc:195] TabletServer@127.31.48.129:33149 shutdown complete.
I20250905 02:22:55.217187 31938 tablet_server.cc:178] TabletServer@127.31.48.130:36981 shutting down...
I20250905 02:22:55.241447 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:55.242177 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.242980 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 3 LEADER]: Raft consensus shutting down.
I20250905 02:22:55.244037 31938 pending_rounds.cc:70] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Trying to abort 1 pending ops.
I20250905 02:22:55.244338 31938 pending_rounds.cc:77] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1: Aborting op as it isn't in flight: id { term: 3 index: 3 } timestamp: 7196831641036836864 op_type: NO_OP noop_request { }
I20250905 02:22:55.244776 31938 raft_consensus.cc:2887] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 3 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:22:55.245137 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.247956 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.248739 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:55.249332 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.252128 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.252918 31938 raft_consensus.cc:2241] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:55.253432 31938 raft_consensus.cc:2270] T 78febd0e466543a2a51dab418bb9e9c6 P 6c7070b76dbc426784bf84a6040700d1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:55.255676 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:55.256320 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:55.256893 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P 6c7070b76dbc426784bf84a6040700d1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 02:22:56.175467 4218 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
I20250905 02:22:56.176312 4897 raft_consensus.cc:491] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 54835050b80544d880704fa9291f1905)
I20250905 02:22:56.176806 4897 raft_consensus.cc:513] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:56.178627 4897 leader_election.cc:290] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
W20250905 02:22:56.197305 4757 leader_election.cc:336] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149): Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.198065 4756 leader_election.cc:336] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981): Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
I20250905 02:22:56.198527 4756 leader_election.cc:304] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b; no voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1
I20250905 02:22:56.199354 4897 raft_consensus.cc:2747] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 02:22:56.200395 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
W20250905 02:22:56.200724 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
W20250905 02:22:56.206010 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.206291 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.210879 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.211139 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.269223 31938 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 4738)
I20250905 02:22:56.285773 4897 raft_consensus.cc:491] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 54835050b80544d880704fa9291f1905)
I20250905 02:22:56.286453 4897 raft_consensus.cc:513] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:56.288743 4897 leader_election.cc:290] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
W20250905 02:22:56.292531 4757 leader_election.cc:336] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149): Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.293062 4756 leader_election.cc:336] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981): Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
I20250905 02:22:56.293546 4756 leader_election.cc:304] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b; no voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1
I20250905 02:22:56.294312 4897 raft_consensus.cc:2747] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 02:22:56.324303 4897 raft_consensus.cc:491] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 54835050b80544d880704fa9291f1905)
I20250905 02:22:56.324853 4897 raft_consensus.cc:513] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } }
I20250905 02:22:56.326604 4897 leader_election.cc:290] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
W20250905 02:22:56.330312 4757 leader_election.cc:336] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149): Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:56.330780 4756 leader_election.cc:336] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981): Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
I20250905 02:22:56.331132 4756 leader_election.cc:304] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b; no voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1
I20250905 02:22:56.331701 4897 raft_consensus.cc:2747] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 02:22:56.917236 4902 raft_consensus.cc:491] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:22:56.917851 4902 raft_consensus.cc:513] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:56.920156 4902 leader_election.cc:290] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981), 54835050b80544d880704fa9291f1905 (127.31.48.129:33149)
W20250905 02:22:56.926789 4756 leader_election.cc:336] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981): Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
W20250905 02:22:56.927459 4757 leader_election.cc:336] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149): Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
I20250905 02:22:56.927809 4757 leader_election.cc:304] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b; no voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1
I20250905 02:22:56.928457 4902 raft_consensus.cc:2747] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 02:22:57.217492 4218 auto_rebalancer.cc:249] could not retrieve auto-rebalancing replica moves: Not found: table d2ae6f32b5c34a0b961ab2764362db50: could not find any suitable replica to move from server 54835050b80544d880704fa9291f1905 to server c41710f0cfb64170a41bc535f47e8fa6
W20250905 02:22:57.269729 31938 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 4738)
W20250905 02:22:58.232159 4218 auto_rebalancer.cc:254] failed to send replica move request: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:58.237660 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:58.237979 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
I20250905 02:22:58.239557 4908 raft_consensus.cc:491] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 54835050b80544d880704fa9291f1905)
I20250905 02:22:58.240097 4908 raft_consensus.cc:513] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } attrs { replace: true } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:58.242064 4908 leader_election.cc:290] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
W20250905 02:22:58.245690 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:58.245993 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
W20250905 02:22:58.249534 4756 leader_election.cc:336] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981): Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
W20250905 02:22:58.250564 4757 leader_election.cc:336] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149): Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
I20250905 02:22:58.250727 31938 tablet_server.cc:195] TabletServer@127.31.48.130:36981 shutdown complete.
W20250905 02:22:58.250891 4218 auto_rebalancer.cc:663] Could not move replica: Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
W20250905 02:22:58.251240 4218 auto_rebalancer.cc:264] scheduled replica move failed to complete: Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
I20250905 02:22:58.251045 4757 leader_election.cc:304] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b; no voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1
I20250905 02:22:58.251976 4908 raft_consensus.cc:2747] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 02:22:58.262897 31938 tablet_server.cc:178] TabletServer@127.31.48.131:46329 shutting down...
I20250905 02:22:58.272003 4908 raft_consensus.cc:491] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 54835050b80544d880704fa9291f1905)
I20250905 02:22:58.272743 4908 raft_consensus.cc:513] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "54835050b80544d880704fa9291f1905" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 33149 } } peers { permanent_uuid: "6c7070b76dbc426784bf84a6040700d1" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 36981 } } peers { permanent_uuid: "437e86f2e6c646c59fd56f2315f1f81b" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 46329 } attrs { replace: true } } peers { permanent_uuid: "c41710f0cfb64170a41bc535f47e8fa6" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 33179 } attrs { promote: true } }
I20250905 02:22:58.276232 4908 leader_election.cc:290] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 54835050b80544d880704fa9291f1905 (127.31.48.129:33149), 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981)
W20250905 02:22:58.282547 4756 leader_election.cc:336] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c7070b76dbc426784bf84a6040700d1 (127.31.48.130:36981): Network error: Client connection negotiation failed: client connection to 127.31.48.130:36981: connect: Connection refused (error 111)
W20250905 02:22:58.284140 4757 leader_election.cc:336] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 54835050b80544d880704fa9291f1905 (127.31.48.129:33149): Network error: Client connection negotiation failed: client connection to 127.31.48.129:33149: connect: Connection refused (error 111)
I20250905 02:22:58.284773 4757 leader_election.cc:304] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 437e86f2e6c646c59fd56f2315f1f81b; no voters: 54835050b80544d880704fa9291f1905, 6c7070b76dbc426784bf84a6040700d1
I20250905 02:22:58.286280 4908 raft_consensus.cc:2747] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 02:22:58.299234 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:58.300079 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:58.300926 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:58.301781 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:58.304512 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:58.305202 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:58.305815 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:58.308099 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:58.308795 31938 raft_consensus.cc:2241] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:58.309334 31938 raft_consensus.cc:2270] T 78febd0e466543a2a51dab418bb9e9c6 P 437e86f2e6c646c59fd56f2315f1f81b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:58.311667 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:22:58.312291 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:22:58.312850 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P 437e86f2e6c646c59fd56f2315f1f81b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:58.337160 31938 tablet_server.cc:195] TabletServer@127.31.48.131:46329 shutdown complete.
I20250905 02:22:58.349522 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:22:58.376305 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:22:58.377337 31938 tablet_replica.cc:331] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6: stopping tablet replica
I20250905 02:22:58.378005 31938 raft_consensus.cc:2241] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Raft consensus shutting down.
I20250905 02:22:58.378410 31938 raft_consensus.cc:2270] T b922492bc53645bf90e75036b5aa3a8a P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Raft consensus is shut down!
I20250905 02:22:58.380210 31938 tablet_replica.cc:331] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6: stopping tablet replica
I20250905 02:22:58.381109 31938 raft_consensus.cc:2241] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6 [term 3 LEARNER]: Raft consensus shutting down.
I20250905 02:22:58.381613 31938 raft_consensus.cc:2270] T 691922bedd744294a3219a5605970305 P c41710f0cfb64170a41bc535f47e8fa6 [term 3 LEARNER]: Raft consensus is shut down!
I20250905 02:22:58.383373 31938 tablet_replica.cc:331] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6: stopping tablet replica
I20250905 02:22:58.383857 31938 raft_consensus.cc:2241] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Raft consensus shutting down.
I20250905 02:22:58.384225 31938 raft_consensus.cc:2270] T 1745e6d53a45442c91d58026e7c5c0c1 P c41710f0cfb64170a41bc535f47e8fa6 [term 2 LEARNER]: Raft consensus is shut down!
I20250905 02:22:58.419817 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:22:58.431157 31938 master.cc:561] Master@127.31.48.190:34765 shutting down...
I20250905 02:22:58.450279 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:22:58.450837 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:22:58.451185 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 7727fa2a20e84a9d999069929f8b176b: stopping tablet replica
I20250905 02:22:58.470937 31938 master.cc:583] Master@127.31.48.190:34765 shutdown complete.
[ OK ] AutoRebalancerTest.TestHandlingFailedTservers (15884 ms)
[ RUN ] AutoRebalancerTest.TestDeletedTables
I20250905 02:22:58.505343 31938 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.31.48.190:40685
I20250905 02:22:58.506394 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:58.511448 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:58.512166 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.513386 31938 server_base.cc:1047] running on GCE node
W20250905 02:22:58.513087 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.514601 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:58.514781 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:58.514946 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038978514926 us; error 0 us; skew 500 ppm
I20250905 02:22:58.515424 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:58.517717 31938 webserver.cc:480] Webserver started at http://127.31.48.190:37693/ using document root <none> and password file <none>
I20250905 02:22:58.518179 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:58.518344 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:58.518602 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:58.519639 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/master-0-root/instance:
uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.523911 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20250905 02:22:58.527221 4920 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.527956 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250905 02:22:58.528226 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/master-0-root
uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.528537 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:58.544328 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:58.545491 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:58.581226 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.190:40685
I20250905 02:22:58.581326 4971 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.190:40685 every 8 connection(s)
I20250905 02:22:58.584848 4972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:22:58.594810 4972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Bootstrap starting.
I20250905 02:22:58.599184 4972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Neither blocks nor log segments found. Creating new log.
I20250905 02:22:58.603080 4972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: No bootstrap required, opened a new log
I20250905 02:22:58.605093 4972 raft_consensus.cc:357] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" member_type: VOTER }
I20250905 02:22:58.605631 4972 raft_consensus.cc:383] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:22:58.605841 4972 raft_consensus.cc:738] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ab9b1d7ea9450f9d4fa09bd5afa1bc, State: Initialized, Role: FOLLOWER
I20250905 02:22:58.606314 4972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" member_type: VOTER }
I20250905 02:22:58.606716 4972 raft_consensus.cc:397] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 02:22:58.606897 4972 raft_consensus.cc:491] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 02:22:58.607124 4972 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:22:58.611450 4972 raft_consensus.cc:513] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" member_type: VOTER }
I20250905 02:22:58.611900 4972 leader_election.cc:304] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83ab9b1d7ea9450f9d4fa09bd5afa1bc; no voters:
I20250905 02:22:58.612926 4972 leader_election.cc:290] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 02:22:58.613320 4975 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:22:58.614578 4975 raft_consensus.cc:695] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 1 LEADER]: Becoming Leader. State: Replica: 83ab9b1d7ea9450f9d4fa09bd5afa1bc, State: Running, Role: LEADER
I20250905 02:22:58.615111 4975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" member_type: VOTER }
I20250905 02:22:58.615626 4972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [sys.catalog]: configured and running, proceeding with master startup.
I20250905 02:22:58.619217 4976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" member_type: VOTER } }
I20250905 02:22:58.619809 4976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:58.621403 4977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83ab9b1d7ea9450f9d4fa09bd5afa1bc. Latest consensus state: current_term: 1 leader_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83ab9b1d7ea9450f9d4fa09bd5afa1bc" member_type: VOTER } }
I20250905 02:22:58.621832 4981 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 02:22:58.621966 4977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [sys.catalog]: This master's current role is: LEADER
I20250905 02:22:58.630257 4981 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 02:22:58.631314 31938 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20250905 02:22:58.639015 4981 catalog_manager.cc:1349] Generated new cluster ID: 6e72d314a205439b80a29ab45c047a9e
I20250905 02:22:58.639285 4981 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 02:22:58.662745 4981 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 02:22:58.664026 4981 catalog_manager.cc:1506] Loading token signing keys...
I20250905 02:22:58.684512 4981 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Generated new TSK 0
I20250905 02:22:58.685081 4981 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 02:22:58.697548 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:58.703179 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:58.704423 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.705829 31938 server_base.cc:1047] running on GCE node
W20250905 02:22:58.706135 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.707783 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:58.708003 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:58.708163 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038978708145 us; error 0 us; skew 500 ppm
I20250905 02:22:58.708673 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:58.710734 31938 webserver.cc:480] Webserver started at http://127.31.48.129:42497/ using document root <none> and password file <none>
I20250905 02:22:58.711184 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:58.711369 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:58.711606 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:58.712769 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/instance:
uuid: "92b5c3bc59924a5f88755f068a0dea18"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.717051 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 02:22:58.720042 5001 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.720750 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250905 02:22:58.721000 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "92b5c3bc59924a5f88755f068a0dea18"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.721256 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:58.746155 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:58.747339 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:58.748818 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:58.750937 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:58.751127 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.751368 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:58.751520 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.789342 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:35789
I20250905 02:22:58.789427 5063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:35789 every 8 connection(s)
I20250905 02:22:58.793690 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:58.800722 5068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:58.802578 5069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:58.803881 5071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.805390 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:58.806294 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20250905 02:22:58.806437 5064 heartbeater.cc:344] Connected to a master server at 127.31.48.190:40685
W20250905 02:22:58.806535 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:58.806847 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038978806828 us; error 0 us; skew 500 ppm
I20250905 02:22:58.806895 5064 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:58.807382 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:58.807688 5064 heartbeater.cc:507] Master 127.31.48.190:40685 requested a full tablet report, sending...
I20250905 02:22:58.810186 31938 webserver.cc:480] Webserver started at http://127.31.48.130:46301/ using document root <none> and password file <none>
I20250905 02:22:58.810359 4937 ts_manager.cc:194] Registered new tserver with Master: 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789)
I20250905 02:22:58.810774 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:58.810992 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:58.811244 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:58.812258 4937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48086
I20250905 02:22:58.812288 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root/instance:
uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.816843 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250905 02:22:58.819808 5076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.820585 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250905 02:22:58.820815 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root
uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.821031 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:58.837869 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:58.839021 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:58.840305 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:58.842432 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:58.842610 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.842823 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:58.842957 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.879216 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.130:37139
I20250905 02:22:58.879364 5138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.130:37139 every 8 connection(s)
I20250905 02:22:58.883970 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 02:22:58.890149 5142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:22:58.891963 5143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.894627 31938 server_base.cc:1047] running on GCE node
I20250905 02:22:58.895547 5139 heartbeater.cc:344] Connected to a master server at 127.31.48.190:40685
W20250905 02:22:58.895620 5145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:22:58.896036 5139 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:58.896685 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:22:58.896965 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:22:58.896950 5139 heartbeater.cc:507] Master 127.31.48.190:40685 requested a full tablet report, sending...
I20250905 02:22:58.897365 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038978897345 us; error 0 us; skew 500 ppm
I20250905 02:22:58.898072 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:22:58.899197 4937 ts_manager.cc:194] Registered new tserver with Master: 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:22:58.900489 31938 webserver.cc:480] Webserver started at http://127.31.48.131:37361/ using document root <none> and password file <none>
I20250905 02:22:58.900709 4937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48102
I20250905 02:22:58.900971 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:22:58.901213 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:22:58.901468 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:22:58.902809 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/instance:
uuid: "2ef8e002be104043af6666699b5396eb"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.907018 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250905 02:22:58.910122 5150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.910826 31938 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20250905 02:22:58.911103 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root
uuid: "2ef8e002be104043af6666699b5396eb"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:22:58.911377 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:22:58.932718 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:22:58.933710 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:22:58.935051 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:22:58.937173 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:22:58.937381 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.937609 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:22:58.937764 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:22:58.974885 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.131:42239
I20250905 02:22:58.974959 5212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.131:42239 every 8 connection(s)
I20250905 02:22:58.988055 5213 heartbeater.cc:344] Connected to a master server at 127.31.48.190:40685
I20250905 02:22:58.988466 5213 heartbeater.cc:461] Registering TS with master...
I20250905 02:22:58.989146 5213 heartbeater.cc:507] Master 127.31.48.190:40685 requested a full tablet report, sending...
I20250905 02:22:58.991114 4937 ts_manager.cc:194] Registered new tserver with Master: 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:22:58.991622 31938 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.013385021s
I20250905 02:22:58.992583 4937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48110
I20250905 02:22:59.814905 5064 heartbeater.cc:499] Master 127.31.48.190:40685 was elected leader, sending a full tablet report...
I20250905 02:22:59.903278 5139 heartbeater.cc:499] Master 127.31.48.190:40685 was elected leader, sending a full tablet report...
I20250905 02:22:59.995283 5213 heartbeater.cc:499] Master 127.31.48.190:40685 was elected leader, sending a full tablet report...
I20250905 02:23:00.023631 31938 test_util.cc:276] Using random seed: -2075998920
I20250905 02:23:00.044938 4937 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48124:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:23:00.047080 4937 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:23:00.108031 5178 tablet_service.cc:1468] Processing CreateTablet for tablet f302a9b041cb44408987319aac57e90b (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:23:00.109498 5178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f302a9b041cb44408987319aac57e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.111083 5176 tablet_service.cc:1468] Processing CreateTablet for tablet 27808973f5d24f91bfe2a2fd1495e472 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:23:00.112557 5176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27808973f5d24f91bfe2a2fd1495e472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.112669 5104 tablet_service.cc:1468] Processing CreateTablet for tablet f302a9b041cb44408987319aac57e90b (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:23:00.111172 5177 tablet_service.cc:1468] Processing CreateTablet for tablet 2c0aed9a0934499ab22c6b93c4f9f915 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:23:00.113934 5104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f302a9b041cb44408987319aac57e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.113809 5101 tablet_service.cc:1468] Processing CreateTablet for tablet 4c99ff2310b74e33a06ecc309b900cf8 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:23:00.114606 5177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0aed9a0934499ab22c6b93c4f9f915. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.115792 5102 tablet_service.cc:1468] Processing CreateTablet for tablet 27808973f5d24f91bfe2a2fd1495e472 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:23:00.117061 5102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27808973f5d24f91bfe2a2fd1495e472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.121234 5103 tablet_service.cc:1468] Processing CreateTablet for tablet 2c0aed9a0934499ab22c6b93c4f9f915 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:23:00.121883 5175 tablet_service.cc:1468] Processing CreateTablet for tablet 4c99ff2310b74e33a06ecc309b900cf8 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:23:00.122433 5103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0aed9a0934499ab22c6b93c4f9f915. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.123042 5175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c99ff2310b74e33a06ecc309b900cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.124790 5101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c99ff2310b74e33a06ecc309b900cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.140347 5029 tablet_service.cc:1468] Processing CreateTablet for tablet f302a9b041cb44408987319aac57e90b (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:23:00.141710 5029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f302a9b041cb44408987319aac57e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.145423 5028 tablet_service.cc:1468] Processing CreateTablet for tablet 2c0aed9a0934499ab22c6b93c4f9f915 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:23:00.146716 5028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c0aed9a0934499ab22c6b93c4f9f915. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.149433 5027 tablet_service.cc:1468] Processing CreateTablet for tablet 27808973f5d24f91bfe2a2fd1495e472 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:23:00.149966 5234 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.150697 5027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27808973f5d24f91bfe2a2fd1495e472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.153738 5026 tablet_service.cc:1468] Processing CreateTablet for tablet 4c99ff2310b74e33a06ecc309b900cf8 (DEFAULT_TABLE table=test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:23:00.155267 5026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c99ff2310b74e33a06ecc309b900cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.161304 5234 tablet_bootstrap.cc:654] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.182868 5234 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.183449 5234 ts_tablet_manager.cc:1397] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.015s
I20250905 02:23:00.182790 5236 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.186058 5234 raft_consensus.cc:357] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.186833 5234 raft_consensus.cc:383] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.187155 5234 raft_consensus.cc:738] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.189294 5236 tablet_bootstrap.cc:654] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.191004 5234 consensus_queue.cc:260] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.201179 5234 ts_tablet_manager.cc:1428] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.017s user 0.006s sys 0.010s
I20250905 02:23:00.205134 5236 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:00.205678 5236 ts_tablet_manager.cc:1397] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.005s
I20250905 02:23:00.208082 5234 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.213966 5234 tablet_bootstrap.cc:654] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.213709 5236 raft_consensus.cc:357] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.214977 5236 raft_consensus.cc:383] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.215454 5236 raft_consensus.cc:738] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.216697 5236 consensus_queue.cc:260] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.221743 5233 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.227223 5238 raft_consensus.cc:491] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:00.227376 5236 ts_tablet_manager.cc:1428] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.021s user 0.010s sys 0.005s
I20250905 02:23:00.228055 5234 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.228562 5234 ts_tablet_manager.cc:1397] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20250905 02:23:00.228672 5236 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.229534 5233 tablet_bootstrap.cc:654] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.231272 5234 raft_consensus.cc:357] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.231967 5234 raft_consensus.cc:383] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.232265 5234 raft_consensus.cc:738] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.233085 5234 consensus_queue.cc:260] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.227728 5238 raft_consensus.cc:513] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.236022 5236 tablet_bootstrap.cc:654] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.251662 5238 leader_election.cc:290] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:00.314368 5236 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:00.315011 5236 ts_tablet_manager.cc:1397] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.087s user 0.046s sys 0.038s
I20250905 02:23:00.315601 5039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27808973f5d24f91bfe2a2fd1495e472" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18" is_pre_election: true
I20250905 02:23:00.316309 5234 ts_tablet_manager.cc:1428] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.087s user 0.029s sys 0.054s
I20250905 02:23:00.317387 5234 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
W20250905 02:23:00.317435 5154 leader_election.cc:343] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:23:00.318812 5114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27808973f5d24f91bfe2a2fd1495e472" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:00.320495 5233 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:00.320997 5233 ts_tablet_manager.cc:1397] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.100s user 0.034s sys 0.054s
W20250905 02:23:00.322331 5152 leader_election.cc:343] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:23:00.322906 5152 leader_election.cc:304] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef8e002be104043af6666699b5396eb; no voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:00.323576 5242 raft_consensus.cc:2747] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 02:23:00.317461 5236 raft_consensus.cc:357] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.323895 5233 raft_consensus.cc:357] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.323623 5234 tablet_bootstrap.cc:654] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.324832 5236 raft_consensus.cc:383] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.325001 5233 raft_consensus.cc:383] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.325184 5236 raft_consensus.cc:738] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.325451 5233 raft_consensus.cc:738] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.325920 5236 consensus_queue.cc:260] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.326171 5233 consensus_queue.cc:260] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.328935 5236 ts_tablet_manager.cc:1428] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20250905 02:23:00.330156 5236 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.337836 5236 tablet_bootstrap.cc:654] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.339368 5240 raft_consensus.cc:491] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:00.340085 5240 raft_consensus.cc:513] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.346920 5240 leader_election.cc:290] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.347641 5240 raft_consensus.cc:491] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:00.348266 5240 raft_consensus.cc:513] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.350919 5240 leader_election.cc:290] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:00.353127 5234 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.353806 5234 ts_tablet_manager.cc:1397] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.025s
I20250905 02:23:00.356483 5234 raft_consensus.cc:357] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.357329 5234 raft_consensus.cc:383] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.357700 5234 raft_consensus.cc:738] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.358131 5233 ts_tablet_manager.cc:1428] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.037s user 0.013s sys 0.022s
I20250905 02:23:00.359140 5233 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.358577 5234 consensus_queue.cc:260] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.361356 5234 ts_tablet_manager.cc:1428] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20250905 02:23:00.362318 5234 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.365860 5233 tablet_bootstrap.cc:654] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.367805 5234 tablet_bootstrap.cc:654] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.378804 5234 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.379267 5234 ts_tablet_manager.cc:1397] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250905 02:23:00.382155 5236 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:00.382830 5236 ts_tablet_manager.cc:1397] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.000s
I20250905 02:23:00.381740 5234 raft_consensus.cc:357] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.383589 5234 raft_consensus.cc:383] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.383886 5234 raft_consensus.cc:738] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.384670 5234 consensus_queue.cc:260] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.387339 5236 raft_consensus.cc:357] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.388487 5236 raft_consensus.cc:383] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.388893 5236 raft_consensus.cc:738] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.389163 5234 ts_tablet_manager.cc:1428] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20250905 02:23:00.392701 5233 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:00.393304 5233 ts_tablet_manager.cc:1397] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.012s
I20250905 02:23:00.395957 5114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:00.396334 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:00.396543 5236 consensus_queue.cc:260] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.397106 5113 raft_consensus.cc:2466] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
W20250905 02:23:00.397581 5003 leader_election.cc:343] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:23:00.398326 5003 leader_election.cc:304] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:00.398416 5233 raft_consensus.cc:357] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.399060 5236 ts_tablet_manager.cc:1428] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.016s user 0.007s sys 0.002s
I20250905 02:23:00.399243 5233 raft_consensus.cc:383] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.399449 5240 raft_consensus.cc:2802] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:00.399616 5233 raft_consensus.cc:738] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.399894 5240 raft_consensus.cc:491] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:00.400260 5240 raft_consensus.cc:3058] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.401086 5236 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.401468 5233 consensus_queue.cc:260] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.404582 5233 ts_tablet_manager.cc:1428] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20250905 02:23:00.405603 5233 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.407425 5188 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:00.408023 5236 tablet_bootstrap.cc:654] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.408213 5188 raft_consensus.cc:2466] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:00.408900 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:00.409665 5004 leader_election.cc:304] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18; no voters: 5e5a0b8a2e3e47839db538a0abe12bee
I20250905 02:23:00.409662 5187 raft_consensus.cc:2466] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:00.411511 5233 tablet_bootstrap.cc:654] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.411798 5248 raft_consensus.cc:2802] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:00.412410 5248 raft_consensus.cc:491] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:00.412840 5248 raft_consensus.cc:3058] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.419328 5240 raft_consensus.cc:513] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.435992 5240 leader_election.cc:290] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Requested vote from peers 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:00.437462 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:00.437952 5187 raft_consensus.cc:3058] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.439822 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:00.440497 5113 raft_consensus.cc:3058] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.440659 5236 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:00.441212 5236 ts_tablet_manager.cc:1397] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.008s
I20250905 02:23:00.441885 5233 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:00.442420 5233 ts_tablet_manager.cc:1397] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.037s user 0.022s sys 0.011s
I20250905 02:23:00.444021 5236 raft_consensus.cc:357] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.444797 5236 raft_consensus.cc:383] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.445071 5236 raft_consensus.cc:738] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.445398 5233 raft_consensus.cc:357] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.446175 5233 raft_consensus.cc:383] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.445758 5236 consensus_queue.cc:260] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.446484 5233 raft_consensus.cc:738] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.446545 5187 raft_consensus.cc:2466] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:00.447412 5233 consensus_queue.cc:260] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.448081 5247 raft_consensus.cc:491] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:00.448096 5004 leader_election.cc:304] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:00.423974 5248 raft_consensus.cc:513] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.448909 5247 raft_consensus.cc:513] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.449472 5240 raft_consensus.cc:2802] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:00.450090 5240 raft_consensus.cc:695] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Becoming Leader. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Running, Role: LEADER
I20250905 02:23:00.450700 5248 leader_election.cc:290] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Requested vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.450973 5240 consensus_queue.cc:237] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:00.454703 5233 ts_tablet_manager.cc:1428] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250905 02:23:00.455732 5113 raft_consensus.cc:2466] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:00.456276 5233 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.459430 5247 leader_election.cc:290] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.465480 5233 tablet_bootstrap.cc:654] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.448318 5236 ts_tablet_manager.cc:1428] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250905 02:23:00.470588 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:00.471496 5187 raft_consensus.cc:3058] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.481204 5233 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:00.481899 5233 ts_tablet_manager.cc:1397] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.026s user 0.023s sys 0.000s
I20250905 02:23:00.484768 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:00.484879 5233 raft_consensus.cc:357] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.485947 5233 raft_consensus.cc:383] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.486502 5233 raft_consensus.cc:738] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.485878 4936 catalog_manager.cc:5582] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "92b5c3bc59924a5f88755f068a0dea18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:00.487584 5233 consensus_queue.cc:260] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.490065 5233 ts_tablet_manager.cc:1428] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250905 02:23:00.491277 5113 raft_consensus.cc:3058] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.495136 5187 raft_consensus.cc:2466] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:00.497486 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c99ff2310b74e33a06ecc309b900cf8" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:00.497691 5004 leader_election.cc:304] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:00.498165 5187 raft_consensus.cc:2466] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 0.
I20250905 02:23:00.498817 5240 raft_consensus.cc:2802] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:00.499140 5113 raft_consensus.cc:2466] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:00.499310 5079 leader_election.cc:304] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters:
I20250905 02:23:00.500262 5247 raft_consensus.cc:2802] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:00.500679 5247 raft_consensus.cc:491] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:00.501030 5247 raft_consensus.cc:3058] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.499364 5240 raft_consensus.cc:695] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Becoming Leader. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Running, Role: LEADER
I20250905 02:23:00.502545 5240 consensus_queue.cc:237] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.506881 5039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c99ff2310b74e33a06ecc309b900cf8" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18" is_pre_election: true
I20250905 02:23:00.507634 5039 raft_consensus.cc:2466] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 0.
I20250905 02:23:00.508993 5247 raft_consensus.cc:513] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.511763 5039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c99ff2310b74e33a06ecc309b900cf8" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18"
I20250905 02:23:00.512004 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c99ff2310b74e33a06ecc309b900cf8" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:00.512396 5039 raft_consensus.cc:3058] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.512552 5187 raft_consensus.cc:3058] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.516999 5247 leader_election.cc:290] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 election: Requested vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.517475 5187 raft_consensus.cc:2466] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 1.
I20250905 02:23:00.517695 5039 raft_consensus.cc:2466] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 1.
I20250905 02:23:00.518458 5079 leader_election.cc:304] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters:
I20250905 02:23:00.519229 5247 raft_consensus.cc:2802] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:00.519676 5247 raft_consensus.cc:695] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Becoming Leader. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Running, Role: LEADER
I20250905 02:23:00.519941 4936 catalog_manager.cc:5582] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "92b5c3bc59924a5f88755f068a0dea18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:00.520303 5247 consensus_queue.cc:237] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.528625 4937 catalog_manager.cc:5582] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130). New cstate: current_term: 1 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:00.592974 5247 raft_consensus.cc:491] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:00.593480 5247 raft_consensus.cc:513] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.595018 5247 leader_election.cc:290] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.595994 5039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27808973f5d24f91bfe2a2fd1495e472" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18" is_pre_election: true
I20250905 02:23:00.596136 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27808973f5d24f91bfe2a2fd1495e472" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:00.596691 5039 raft_consensus.cc:2466] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 0.
I20250905 02:23:00.596781 5187 raft_consensus.cc:2466] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 0.
I20250905 02:23:00.597721 5080 leader_election.cc:304] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:00.598541 5247 raft_consensus.cc:2802] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:00.598827 5247 raft_consensus.cc:491] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:00.599076 5247 raft_consensus.cc:3058] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.603729 5247 raft_consensus.cc:513] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.605279 5247 leader_election.cc:290] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 election: Requested vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.606146 5039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27808973f5d24f91bfe2a2fd1495e472" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18"
I20250905 02:23:00.606226 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27808973f5d24f91bfe2a2fd1495e472" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:00.606632 5039 raft_consensus.cc:3058] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.606724 5187 raft_consensus.cc:3058] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.611485 5187 raft_consensus.cc:2466] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 1.
I20250905 02:23:00.612015 5039 raft_consensus.cc:2466] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 1.
I20250905 02:23:00.612541 5079 leader_election.cc:304] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters:
I20250905 02:23:00.613320 5247 raft_consensus.cc:2802] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:00.613693 5247 raft_consensus.cc:695] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Becoming Leader. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Running, Role: LEADER
I20250905 02:23:00.614425 5247 consensus_queue.cc:237] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.620512 4937 catalog_manager.cc:5582] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130). New cstate: current_term: 1 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:00.709782 31938 test_util.cc:276] Using random seed: -2075312776
I20250905 02:23:00.732348 4937 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48128:
name: "dugtrio"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_""\004\001\000\377\377\377\037\004\001\000\376\377\377?\004\001\000\375\377\377_"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 02:23:00.734615 4937 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table dugtrio in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 02:23:00.766571 5026 tablet_service.cc:1468] Processing CreateTablet for tablet 539accf72c2a46dbbe2eae009b09d933 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:23:00.768055 5026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539accf72c2a46dbbe2eae009b09d933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.767755 5104 tablet_service.cc:1468] Processing CreateTablet for tablet 539accf72c2a46dbbe2eae009b09d933 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:23:00.768460 5177 tablet_service.cc:1468] Processing CreateTablet for tablet 539accf72c2a46dbbe2eae009b09d933 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION VALUES < 536870911
I20250905 02:23:00.769322 5104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539accf72c2a46dbbe2eae009b09d933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.769670 5177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539accf72c2a46dbbe2eae009b09d933. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.770450 5103 tablet_service.cc:1468] Processing CreateTablet for tablet 1243692508e84769a132c2535f5d1c98 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:23:00.771674 5103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1243692508e84769a132c2535f5d1c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.782353 5027 tablet_service.cc:1468] Processing CreateTablet for tablet 1243692508e84769a132c2535f5d1c98 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:23:00.783730 5027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1243692508e84769a132c2535f5d1c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.784742 5236 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.784957 5102 tablet_service.cc:1468] Processing CreateTablet for tablet a27c1e35910b4159a54954ff106d89cc (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:23:00.786325 5102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27c1e35910b4159a54954ff106d89cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.787438 5175 tablet_service.cc:1468] Processing CreateTablet for tablet 1243692508e84769a132c2535f5d1c98 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 536870911 <= VALUES < 1073741822
I20250905 02:23:00.788806 5175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1243692508e84769a132c2535f5d1c98. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.790396 5236 tablet_bootstrap.cc:654] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.799283 5236 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:00.799796 5236 ts_tablet_manager.cc:1397] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250905 02:23:00.800295 5233 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.801105 5176 tablet_service.cc:1468] Processing CreateTablet for tablet a27c1e35910b4159a54954ff106d89cc (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:23:00.802496 5176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27c1e35910b4159a54954ff106d89cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.802416 5026 tablet_service.cc:1468] Processing CreateTablet for tablet a27c1e35910b4159a54954ff106d89cc (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 1073741822 <= VALUES < 1610612733
I20250905 02:23:00.802626 5236 raft_consensus.cc:357] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.803546 5236 raft_consensus.cc:383] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.803876 5236 raft_consensus.cc:738] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.804057 5026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27c1e35910b4159a54954ff106d89cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.804843 5236 consensus_queue.cc:260] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.808296 5236 ts_tablet_manager.cc:1428] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20250905 02:23:00.807812 5233 tablet_bootstrap.cc:654] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.816035 5234 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.821642 5236 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.823272 5234 tablet_bootstrap.cc:654] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.836915 5236 tablet_bootstrap.cc:654] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.836217 5029 tablet_service.cc:1468] Processing CreateTablet for tablet 76a3bec2a01b4f58bb3270cd9e576291 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:23:00.842295 5233 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:00.842793 5233 ts_tablet_manager.cc:1397] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.043s user 0.020s sys 0.001s
I20250905 02:23:00.842785 5029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a3bec2a01b4f58bb3270cd9e576291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.845619 5233 raft_consensus.cc:357] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.846448 5233 raft_consensus.cc:383] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.846781 5233 raft_consensus.cc:738] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.847623 5233 consensus_queue.cc:260] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.853685 5233 ts_tablet_manager.cc:1428] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250905 02:23:00.854604 5233 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.855855 5175 tablet_service.cc:1468] Processing CreateTablet for tablet 76a3bec2a01b4f58bb3270cd9e576291 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:23:00.860183 5175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a3bec2a01b4f58bb3270cd9e576291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.862600 5234 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.863308 5234 ts_tablet_manager.cc:1397] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.002s
I20250905 02:23:00.868199 5102 tablet_service.cc:1468] Processing CreateTablet for tablet 76a3bec2a01b4f58bb3270cd9e576291 (DEFAULT_TABLE table=dugtrio [id=f7e87748bb06486588b7a2a39a37a757]), partition=RANGE (key) PARTITION 1610612733 <= VALUES
I20250905 02:23:00.869695 5102 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a3bec2a01b4f58bb3270cd9e576291. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:00.869846 5236 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:00.870501 5236 ts_tablet_manager.cc:1397] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.012s
I20250905 02:23:00.873772 5233 tablet_bootstrap.cc:654] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.873986 5234 raft_consensus.cc:357] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.874893 5234 raft_consensus.cc:383] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.875253 5234 raft_consensus.cc:738] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.876120 5234 consensus_queue.cc:260] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.878806 5234 ts_tablet_manager.cc:1428] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20250905 02:23:00.879854 5234 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.886348 5240 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:23:00.882979 5236 raft_consensus.cc:357] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.887295 5236 raft_consensus.cc:383] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.887645 5236 raft_consensus.cc:738] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.891952 5233 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:00.891750 5236 consensus_queue.cc:260] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.892591 5233 ts_tablet_manager.cc:1397] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.009s
I20250905 02:23:00.892680 5240 raft_consensus.cc:491] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:00.893194 5240 raft_consensus.cc:513] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.894055 5236 ts_tablet_manager.cc:1428] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250905 02:23:00.895099 5236 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:00.895902 5233 raft_consensus.cc:357] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.897255 5233 raft_consensus.cc:383] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.897552 5233 raft_consensus.cc:738] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.898315 5233 consensus_queue.cc:260] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.899302 5234 tablet_bootstrap.cc:654] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.900718 5233 ts_tablet_manager.cc:1428] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20250905 02:23:00.901759 5233 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:00.905507 5236 tablet_bootstrap.cc:654] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.906535 5234 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.907032 5234 ts_tablet_manager.cc:1397] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.003s
I20250905 02:23:00.907842 5233 tablet_bootstrap.cc:654] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.909747 5234 raft_consensus.cc:357] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.910548 5234 raft_consensus.cc:383] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.910866 5234 raft_consensus.cc:738] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.911583 5234 consensus_queue.cc:260] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.911559 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1243692508e84769a132c2535f5d1c98" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:00.931623 5113 raft_consensus.cc:2466] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:00.932972 5003 leader_election.cc:304] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:00.932890 5234 ts_tablet_manager.cc:1428] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250905 02:23:00.933931 5257 raft_consensus.cc:2802] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:00.934322 5257 raft_consensus.cc:491] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:00.934619 5257 raft_consensus.cc:3058] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.934959 5234 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.942018 5234 tablet_bootstrap.cc:654] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.950500 5188 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1243692508e84769a132c2535f5d1c98" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:00.951296 5188 raft_consensus.cc:2466] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:00.952941 5240 leader_election.cc:290] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.953537 5234 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:00.954072 5234 ts_tablet_manager.cc:1397] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.004s
I20250905 02:23:00.959686 5234 raft_consensus.cc:357] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.960572 5234 raft_consensus.cc:383] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:00.960909 5234 raft_consensus.cc:738] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:00.961649 5234 consensus_queue.cc:260] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.964352 5234 ts_tablet_manager.cc:1428] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20250905 02:23:00.965413 5234 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Bootstrap starting.
I20250905 02:23:00.971781 5234 tablet_bootstrap.cc:654] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:00.943761 5257 raft_consensus.cc:513] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:00.973731 5257 leader_election.cc:290] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Requested vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:00.977257 5187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1243692508e84769a132c2535f5d1c98" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:00.978139 5187 raft_consensus.cc:3058] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:00.985560 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1243692508e84769a132c2535f5d1c98" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:00.985869 5247 consensus_queue.cc:1035] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:23:00.986403 5113 raft_consensus.cc:3058] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.006439 5285 raft_consensus.cc:491] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:01.019376 5285 raft_consensus.cc:513] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.021487 5285 leader_election.cc:290] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:01.028414 5114 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539accf72c2a46dbbe2eae009b09d933" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:01.028534 5186 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539accf72c2a46dbbe2eae009b09d933" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:01.029301 5114 raft_consensus.cc:2466] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:01.029397 5186 raft_consensus.cc:2466] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:01.031764 5004 leader_election.cc:304] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:01.032524 5281 consensus_queue.cc:1035] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20250905 02:23:01.035897 5187 raft_consensus.cc:2466] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:01.040383 5113 raft_consensus.cc:2466] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:01.042189 5233 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:01.043936 5233 ts_tablet_manager.cc:1397] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.142s user 0.036s sys 0.082s
I20250905 02:23:01.042725 5003 leader_election.cc:304] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:01.044109 5234 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: No bootstrap required, opened a new log
I20250905 02:23:01.045714 5234 ts_tablet_manager.cc:1397] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Time spent bootstrapping tablet: real 0.081s user 0.019s sys 0.008s
I20250905 02:23:01.045820 5281 raft_consensus.cc:2802] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:01.045920 5240 raft_consensus.cc:2802] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:01.046337 5281 raft_consensus.cc:695] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Becoming Leader. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Running, Role: LEADER
I20250905 02:23:01.046422 5240 raft_consensus.cc:491] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:01.046922 5240 raft_consensus.cc:3058] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.047238 5281 consensus_queue.cc:237] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.048269 5234 raft_consensus.cc:357] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.048914 5234 raft_consensus.cc:383] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:01.049182 5234 raft_consensus.cc:738] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Initialized, Role: FOLLOWER
I20250905 02:23:01.049221 5233 raft_consensus.cc:357] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.050038 5233 raft_consensus.cc:383] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:01.050349 5233 raft_consensus.cc:738] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:01.051173 5233 consensus_queue.cc:260] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.052731 5236 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:01.051216 5234 consensus_queue.cc:260] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.054147 5236 ts_tablet_manager.cc:1397] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.159s user 0.023s sys 0.081s
I20250905 02:23:01.057217 5236 raft_consensus.cc:357] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.058154 5236 raft_consensus.cc:383] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:01.058535 5236 raft_consensus.cc:738] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:01.060846 5234 ts_tablet_manager.cc:1428] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250905 02:23:01.061264 5236 consensus_queue.cc:260] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.063933 4935 catalog_manager.cc:5582] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "92b5c3bc59924a5f88755f068a0dea18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:01.068791 5236 ts_tablet_manager.cc:1428] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20250905 02:23:01.071138 5236 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:01.074615 5233 ts_tablet_manager.cc:1428] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250905 02:23:01.075767 5233 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Bootstrap starting.
I20250905 02:23:01.081948 5233 tablet_bootstrap.cc:654] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:01.084900 5236 tablet_bootstrap.cc:654] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Neither blocks nor log segments found. Creating new log.
I20250905 02:23:01.098026 5281 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:23:01.099213 5257 consensus_queue.cc:1035] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 02:23:01.103178 5257 raft_consensus.cc:491] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:01.103682 5247 consensus_queue.cc:1035] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 02:23:01.103833 5257 raft_consensus.cc:513] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.110332 5185 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a27c1e35910b4159a54954ff106d89cc" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:01.112327 5185 raft_consensus.cc:2466] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:01.107798 5257 leader_election.cc:290] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:01.116612 5004 leader_election.cc:304] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:01.117942 5257 raft_consensus.cc:2802] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:01.120153 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a27c1e35910b4159a54954ff106d89cc" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:01.118485 5257 raft_consensus.cc:491] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:01.121512 5113 raft_consensus.cc:2466] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 0.
I20250905 02:23:01.121219 5257 raft_consensus.cc:3058] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.141407 5233 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: No bootstrap required, opened a new log
I20250905 02:23:01.141880 5233 ts_tablet_manager.cc:1397] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.030s
I20250905 02:23:01.125167 5240 raft_consensus.cc:513] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.129863 5256 consensus_queue.cc:1035] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20250905 02:23:01.159456 5233 raft_consensus.cc:357] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.161705 5233 raft_consensus.cc:383] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:01.162148 5233 raft_consensus.cc:738] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Initialized, Role: FOLLOWER
I20250905 02:23:01.162974 5233 consensus_queue.cc:260] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.166344 5233 ts_tablet_manager.cc:1428] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Time spent starting tablet: real 0.024s user 0.003s sys 0.003s
I20250905 02:23:01.169534 5240 leader_election.cc:290] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Requested vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:01.170710 5188 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539accf72c2a46dbbe2eae009b09d933" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:01.171322 5188 raft_consensus.cc:3058] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.171950 5257 raft_consensus.cc:513] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.174837 5113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a27c1e35910b4159a54954ff106d89cc" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:01.175472 5113 raft_consensus.cc:3058] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.178537 5188 raft_consensus.cc:2466] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:01.179899 5004 leader_election.cc:304] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:01.180857 5240 raft_consensus.cc:2802] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:01.181576 5240 raft_consensus.cc:695] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Becoming Leader. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Running, Role: LEADER
I20250905 02:23:01.182307 5113 raft_consensus.cc:2466] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:01.182461 5240 consensus_queue.cc:237] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.185427 5003 leader_election.cc:304] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18; no voters:
I20250905 02:23:01.186494 5286 raft_consensus.cc:2802] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:01.184206 5257 leader_election.cc:290] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [CANDIDATE]: Term 1 election: Requested vote from peers 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:01.186942 5286 raft_consensus.cc:695] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Becoming Leader. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Running, Role: LEADER
I20250905 02:23:01.188334 5299 consensus_queue.cc:1035] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 02:23:01.190918 5286 consensus_queue.cc:237] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.184469 5188 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a27c1e35910b4159a54954ff106d89cc" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:01.195324 5188 raft_consensus.cc:3058] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.201099 5112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539accf72c2a46dbbe2eae009b09d933" candidate_uuid: "92b5c3bc59924a5f88755f068a0dea18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:01.212639 5188 raft_consensus.cc:2466] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:01.204202 5112 raft_consensus.cc:3058] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.227452 5112 raft_consensus.cc:2466] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92b5c3bc59924a5f88755f068a0dea18 in term 1.
I20250905 02:23:01.224313 5242 raft_consensus.cc:491] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 02:23:01.233068 5242 raft_consensus.cc:513] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.236901 5242 leader_election.cc:290] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:01.238349 5039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a3bec2a01b4f58bb3270cd9e576291" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18" is_pre_election: true
W20250905 02:23:01.239797 5154 leader_election.cc:343] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Illegal state: must be running to vote when last-logged opid is not known
I20250905 02:23:01.244000 5236 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: No bootstrap required, opened a new log
I20250905 02:23:01.244532 5236 ts_tablet_manager.cc:1397] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.174s user 0.013s sys 0.019s
I20250905 02:23:01.246948 5236 raft_consensus.cc:357] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.247623 5236 raft_consensus.cc:383] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 02:23:01.247880 5236 raft_consensus.cc:738] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:01.249011 5112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a3bec2a01b4f58bb3270cd9e576291" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:01.251003 5236 consensus_queue.cc:260] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.254370 5112 raft_consensus.cc:2466] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef8e002be104043af6666699b5396eb in term 0.
I20250905 02:23:01.255900 5236 ts_tablet_manager.cc:1428] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250905 02:23:01.255882 5152 leader_election.cc:304] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters: 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:01.257458 5242 raft_consensus.cc:2802] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 02:23:01.257877 5242 raft_consensus.cc:491] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 02:23:01.258280 5242 raft_consensus.cc:3058] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.262818 4936 catalog_manager.cc:5582] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "92b5c3bc59924a5f88755f068a0dea18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:01.264747 4936 catalog_manager.cc:5582] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129). New cstate: current_term: 1 leader_uuid: "92b5c3bc59924a5f88755f068a0dea18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } }
I20250905 02:23:01.265621 5242 raft_consensus.cc:513] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.267616 5242 leader_election.cc:290] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 election: Requested vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:01.268649 5112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a3bec2a01b4f58bb3270cd9e576291" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:01.268594 5038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a3bec2a01b4f58bb3270cd9e576291" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b5c3bc59924a5f88755f068a0dea18"
I20250905 02:23:01.269287 5112 raft_consensus.cc:3058] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.269402 5038 raft_consensus.cc:3058] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 0 FOLLOWER]: Advancing to term 1
I20250905 02:23:01.274055 5112 raft_consensus.cc:2466] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef8e002be104043af6666699b5396eb in term 1.
I20250905 02:23:01.275193 5152 leader_election.cc:304] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters:
I20250905 02:23:01.276019 5242 raft_consensus.cc:2802] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader election won for term 1
I20250905 02:23:01.276150 5038 raft_consensus.cc:2466] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef8e002be104043af6666699b5396eb in term 1.
I20250905 02:23:01.277192 5242 raft_consensus.cc:695] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 1 LEADER]: Becoming Leader. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Running, Role: LEADER
I20250905 02:23:01.278432 5242 consensus_queue.cc:237] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:01.286341 4936 catalog_manager.cc:5582] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2ef8e002be104043af6666699b5396eb (127.31.48.131). New cstate: current_term: 1 leader_uuid: "2ef8e002be104043af6666699b5396eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: HEALTHY } } }
I20250905 02:23:01.375610 31938 tablet_server.cc:178] TabletServer@127.31.48.129:0 shutting down...
I20250905 02:23:01.393415 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:23:01.394151 31938 tablet_replica.cc:331] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.394778 31938 raft_consensus.cc:2241] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:01.395336 31938 pending_rounds.cc:70] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Trying to abort 1 pending ops.
I20250905 02:23:01.395498 31938 pending_rounds.cc:77] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831666379538432 op_type: NO_OP noop_request { }
I20250905 02:23:01.395726 31938 raft_consensus.cc:2887] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:23:01.395936 31938 raft_consensus.cc:2270] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.398147 31938 tablet_replica.cc:331] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.398635 31938 raft_consensus.cc:2241] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:01.399119 31938 pending_rounds.cc:70] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Trying to abort 1 pending ops.
I20250905 02:23:01.399263 31938 pending_rounds.cc:77] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831666939293696 op_type: NO_OP noop_request { }
I20250905 02:23:01.399549 31938 raft_consensus.cc:2887] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:23:01.399801 31938 raft_consensus.cc:2270] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.401831 31938 tablet_replica.cc:331] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.402310 31938 raft_consensus.cc:2241] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:01.402776 31938 pending_rounds.cc:70] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Trying to abort 1 pending ops.
I20250905 02:23:01.402915 31938 pending_rounds.cc:77] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831666967597056 op_type: NO_OP noop_request { }
I20250905 02:23:01.403123 31938 raft_consensus.cc:2887] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:23:01.403332 31938 raft_consensus.cc:2270] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.405498 31938 tablet_replica.cc:331] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.405959 31938 raft_consensus.cc:2241] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:01.406301 31938 raft_consensus.cc:2270] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.408274 31938 tablet_replica.cc:331] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.408849 31938 raft_consensus.cc:2241] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:01.409549 31938 raft_consensus.cc:2270] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.411494 31938 tablet_replica.cc:331] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.412003 31938 raft_consensus.cc:2241] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:01.412436 31938 raft_consensus.cc:2270] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.414507 31938 tablet_replica.cc:331] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.414939 31938 raft_consensus.cc:2241] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:01.415247 31938 raft_consensus.cc:2270] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.416970 31938 tablet_replica.cc:331] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: stopping tablet replica
I20250905 02:23:01.417471 31938 raft_consensus.cc:2241] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:01.418152 31938 raft_consensus.cc:2270] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:01.437134 31938 tablet_server.cc:195] TabletServer@127.31.48.129:0 shutdown complete.
W20250905 02:23:01.707999 5080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20250905 02:23:01.712184 5080 consensus_peers.cc:489] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Couldn't send request to peer 92b5c3bc59924a5f88755f068a0dea18. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250905 02:23:01.715564 5080 consensus_peers.cc:489] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Couldn't send request to peer 92b5c3bc59924a5f88755f068a0dea18. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:01.748332 5242 consensus_queue.cc:1035] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 02:23:01.768784 5154 consensus_peers.cc:489] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb -> Peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Couldn't send request to peer 92b5c3bc59924a5f88755f068a0dea18. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:02.467353 4937 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48124:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250905 02:23:02.467952 4937 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48124:
table { table_name: "dugtrio" } modify_external_catalogs: true
I20250905 02:23:02.486250 4937 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Sending DeleteTablet for 3 replicas of tablet 539accf72c2a46dbbe2eae009b09d933
I20250905 02:23:02.487766 4937 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Sending DeleteTablet for 3 replicas of tablet 1243692508e84769a132c2535f5d1c98
I20250905 02:23:02.488931 5102 tablet_service.cc:1515] Processing DeleteTablet for tablet 539accf72c2a46dbbe2eae009b09d933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41380
I20250905 02:23:02.489336 4937 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Sending DeleteTablet for 3 replicas of tablet a27c1e35910b4159a54954ff106d89cc
I20250905 02:23:02.489602 5175 tablet_service.cc:1515] Processing DeleteTablet for tablet 539accf72c2a46dbbe2eae009b09d933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41564
I20250905 02:23:02.490768 4937 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: Sending DeleteTablet for 3 replicas of tablet 76a3bec2a01b4f58bb3270cd9e576291
I20250905 02:23:02.492763 5104 tablet_service.cc:1515] Processing DeleteTablet for tablet a27c1e35910b4159a54954ff106d89cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41380
I20250905 02:23:02.492285 5103 tablet_service.cc:1515] Processing DeleteTablet for tablet 1243692508e84769a132c2535f5d1c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41380
I20250905 02:23:02.494257 5103 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a3bec2a01b4f58bb3270cd9e576291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41380
I20250905 02:23:02.494588 5176 tablet_service.cc:1515] Processing DeleteTablet for tablet 1243692508e84769a132c2535f5d1c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41564
I20250905 02:23:02.495793 5177 tablet_service.cc:1515] Processing DeleteTablet for tablet a27c1e35910b4159a54954ff106d89cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41564
I20250905 02:23:02.496901 5178 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a3bec2a01b4f58bb3270cd9e576291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41564
I20250905 02:23:02.497392 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:23:02.506434 5324 tablet_replica.cc:331] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:02.507548 5325 tablet_replica.cc:331] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:02.508343 5324 raft_consensus.cc:2241] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:02.510128 5247 consensus_queue.cc:579] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Leader has been unable to successfully communicate with peer 92b5c3bc59924a5f88755f068a0dea18 for more than 1 seconds (1.256s)
I20250905 02:23:02.511047 5324 raft_consensus.cc:2270] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.512223 5325 raft_consensus.cc:2241] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 02:23:02.512883 5326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:02.513491 5325 raft_consensus.cc:2270] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.515940 5324 ts_tablet_manager.cc:1905] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:02.529758 5325 ts_tablet_manager.cc:1905] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 02:23:02.530531 4924 catalog_manager.cc:4671] TS 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 539accf72c2a46dbbe2eae009b09d933: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
W20250905 02:23:02.533774 5327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:02.534825 5242 raft_consensus.cc:491] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92b5c3bc59924a5f88755f068a0dea18)
I20250905 02:23:02.535666 5242 raft_consensus.cc:513] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:02.539099 31938 server_base.cc:1047] running on GCE node
W20250905 02:23:02.539125 5331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:02.541033 5242 leader_election.cc:290] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:02.541421 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:23:02.541759 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:23:02.541993 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038982541971 us; error 0 us; skew 500 ppm
I20250905 02:23:02.542877 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:02.542797 5112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:02.548202 5324 ts_tablet_manager.cc:1918] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250905 02:23:02.548646 5154 leader_election.cc:336] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
I20250905 02:23:02.548889 5324 log.cc:1199] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root/wals/539accf72c2a46dbbe2eae009b09d933
I20250905 02:23:02.549185 5154 leader_election.cc:304] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ef8e002be104043af6666699b5396eb; no voters: 5e5a0b8a2e3e47839db538a0abe12bee, 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:02.549788 5324 ts_tablet_manager.cc:1939] T 539accf72c2a46dbbe2eae009b09d933 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting consensus metadata
I20250905 02:23:02.550278 5242 raft_consensus.cc:2747] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 02:23:02.554260 5324 tablet_replica.cc:331] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:02.554404 4922 catalog_manager.cc:4928] TS 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139): tablet 539accf72c2a46dbbe2eae009b09d933 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.555383 5324 raft_consensus.cc:2241] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:02.555945 5324 raft_consensus.cc:2270] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.557049 31938 webserver.cc:480] Webserver started at http://127.31.48.132:45509/ using document root <none> and password file <none>
I20250905 02:23:02.557837 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:02.558144 31938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 02:23:02.558579 31938 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 02:23:02.558827 5324 ts_tablet_manager.cc:1905] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:02.560436 31938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root/instance:
uuid: "50c6b99aa73c43dbae9f577b0c34ae5b"
format_stamp: "Formatted at 2025-09-05 02:23:02 on dist-test-slave-q32q"
I20250905 02:23:02.561724 5325 ts_tablet_manager.cc:1918] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 02:23:02.562273 5325 log.cc:1199] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/wals/539accf72c2a46dbbe2eae009b09d933
I20250905 02:23:02.564497 5325 ts_tablet_manager.cc:1939] T 539accf72c2a46dbbe2eae009b09d933 P 2ef8e002be104043af6666699b5396eb: Deleting consensus metadata
I20250905 02:23:02.573022 4923 catalog_manager.cc:4928] TS 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239): tablet 539accf72c2a46dbbe2eae009b09d933 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.573972 5324 ts_tablet_manager.cc:1918] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 02:23:02.573854 31938 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.006s sys 0.005s
I20250905 02:23:02.573956 5325 tablet_replica.cc:331] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:02.574570 5324 log.cc:1199] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root/wals/a27c1e35910b4159a54954ff106d89cc
I20250905 02:23:02.575363 5324 ts_tablet_manager.cc:1939] T a27c1e35910b4159a54954ff106d89cc P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting consensus metadata
I20250905 02:23:02.575491 5325 raft_consensus.cc:2241] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:02.576246 5325 raft_consensus.cc:2270] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.579367 5324 tablet_replica.cc:331] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:02.579391 5335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.579787 4922 catalog_manager.cc:4928] TS 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139): tablet a27c1e35910b4159a54954ff106d89cc (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.580224 5325 ts_tablet_manager.cc:1905] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:02.580708 5324 raft_consensus.cc:2241] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:02.581076 31938 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250905 02:23:02.581426 5324 raft_consensus.cc:2270] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.581421 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root
uuid: "50c6b99aa73c43dbae9f577b0c34ae5b"
format_stamp: "Formatted at 2025-09-05 02:23:02 on dist-test-slave-q32q"
I20250905 02:23:02.582091 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:02.584862 5324 ts_tablet_manager.cc:1905] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:02.598846 5325 ts_tablet_manager.cc:1918] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 02:23:02.599360 5325 log.cc:1199] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/wals/1243692508e84769a132c2535f5d1c98
I20250905 02:23:02.600059 5325 ts_tablet_manager.cc:1939] T 1243692508e84769a132c2535f5d1c98 P 2ef8e002be104043af6666699b5396eb: Deleting consensus metadata
I20250905 02:23:02.601322 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:02.602346 5324 ts_tablet_manager.cc:1918] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 02:23:02.602861 5324 log.cc:1199] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root/wals/1243692508e84769a132c2535f5d1c98
I20250905 02:23:02.602875 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:02.603662 5324 ts_tablet_manager.cc:1939] T 1243692508e84769a132c2535f5d1c98 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting consensus metadata
I20250905 02:23:02.603803 5325 tablet_replica.cc:331] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:02.603936 4923 catalog_manager.cc:4928] TS 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239): tablet 1243692508e84769a132c2535f5d1c98 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.604833 5325 raft_consensus.cc:2241] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:02.605331 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:02.605747 5325 raft_consensus.cc:2270] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.607467 5324 tablet_replica.cc:331] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:02.607664 4922 catalog_manager.cc:4928] TS 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139): tablet 1243692508e84769a132c2535f5d1c98 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.608647 5324 raft_consensus.cc:2241] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:02.609292 5325 ts_tablet_manager.cc:1905] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:02.609345 5324 raft_consensus.cc:2270] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.609611 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 02:23:02.609961 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.610296 31938 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 02:23:02.610606 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.613330 5324 ts_tablet_manager.cc:1905] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:02.621449 5256 raft_consensus.cc:491] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92b5c3bc59924a5f88755f068a0dea18)
I20250905 02:23:02.622289 5256 raft_consensus.cc:513] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:02.626718 5303 raft_consensus.cc:491] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92b5c3bc59924a5f88755f068a0dea18)
I20250905 02:23:02.627597 5303 raft_consensus.cc:513] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.627684 5325 ts_tablet_manager.cc:1918] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 02:23:02.629256 5325 log.cc:1199] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/wals/a27c1e35910b4159a54954ff106d89cc
I20250905 02:23:02.629985 5325 ts_tablet_manager.cc:1939] T a27c1e35910b4159a54954ff106d89cc P 2ef8e002be104043af6666699b5396eb: Deleting consensus metadata
I20250905 02:23:02.633886 5325 tablet_replica.cc:331] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:02.634795 5325 raft_consensus.cc:2241] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:02.634850 5324 ts_tablet_manager.cc:1918] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 02:23:02.635435 5324 log.cc:1199] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-1-root/wals/76a3bec2a01b4f58bb3270cd9e576291
I20250905 02:23:02.636232 5324 ts_tablet_manager.cc:1939] T 76a3bec2a01b4f58bb3270cd9e576291 P 5e5a0b8a2e3e47839db538a0abe12bee: Deleting consensus metadata
I20250905 02:23:02.636138 5325 raft_consensus.cc:2270] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:02.636612 4923 catalog_manager.cc:4928] TS 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239): tablet a27c1e35910b4159a54954ff106d89cc (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.641321 5186 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
I20250905 02:23:02.643579 4922 catalog_manager.cc:4928] TS 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139): tablet 76a3bec2a01b4f58bb3270cd9e576291 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.644942 5325 ts_tablet_manager.cc:1905] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 02:23:02.644942 5080 leader_election.cc:336] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
I20250905 02:23:02.646651 5080 leader_election.cc:304] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e5a0b8a2e3e47839db538a0abe12bee; no voters: 2ef8e002be104043af6666699b5396eb, 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:02.647449 5247 raft_consensus.cc:2747] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 02:23:02.648156 5080 leader_election.cc:336] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
I20250905 02:23:02.649216 5256 leader_election.cc:290] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:02.638590 5303 leader_election.cc:290] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:02.650350 5186 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb" is_pre_election: true
W20250905 02:23:02.650779 4985 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 92b5c3bc59924a5f88755f068a0dea18 not available for placement
I20250905 02:23:02.651042 5186 raft_consensus.cc:2466] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 1.
I20250905 02:23:02.652662 5079 leader_election.cc:304] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters: 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:02.654397 5303 raft_consensus.cc:2802] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:23:02.654933 5303 raft_consensus.cc:491] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92b5c3bc59924a5f88755f068a0dea18)
I20250905 02:23:02.655613 5303 raft_consensus.cc:3058] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:02.663945 5325 ts_tablet_manager.cc:1918] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 02:23:02.664505 5325 log.cc:1199] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/wals/76a3bec2a01b4f58bb3270cd9e576291
I20250905 02:23:02.665303 5325 ts_tablet_manager.cc:1939] T 76a3bec2a01b4f58bb3270cd9e576291 P 2ef8e002be104043af6666699b5396eb: Deleting consensus metadata
I20250905 02:23:02.666062 5303 raft_consensus.cc:513] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:02.670126 4923 catalog_manager.cc:4928] TS 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239): tablet 76a3bec2a01b4f58bb3270cd9e576291 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:02.671183 5186 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f302a9b041cb44408987319aac57e90b" candidate_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2ef8e002be104043af6666699b5396eb"
I20250905 02:23:02.671872 5303 leader_election.cc:290] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 election: Requested vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239)
I20250905 02:23:02.671995 5186 raft_consensus.cc:3058] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Advancing to term 2
W20250905 02:23:02.673689 5080 leader_election.cc:336] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
I20250905 02:23:02.678462 5186 raft_consensus.cc:2466] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5a0b8a2e3e47839db538a0abe12bee in term 2.
I20250905 02:23:02.679844 5079 leader_election.cc:304] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters: 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:02.680836 5303 raft_consensus.cc:2802] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:23:02.681747 5303 raft_consensus.cc:695] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 LEADER]: Becoming Leader. State: Replica: 5e5a0b8a2e3e47839db538a0abe12bee, State: Running, Role: LEADER
I20250905 02:23:02.682787 5303 consensus_queue.cc:237] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:02.693032 4937 catalog_manager.cc:5582] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee reported cstate change: term changed from 1 to 2, leader changed from 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129) to 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130). New cstate: current_term: 2 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: HEALTHY } } }
I20250905 02:23:02.711551 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.132:40795
I20250905 02:23:02.711760 5398 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.132:40795 every 8 connection(s)
I20250905 02:23:02.716586 31938 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 02:23:02.720736 5242 raft_consensus.cc:491] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 92b5c3bc59924a5f88755f068a0dea18)
I20250905 02:23:02.721503 5242 raft_consensus.cc:513] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
W20250905 02:23:02.722770 5402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:02.725596 5112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" is_pre_election: true
I20250905 02:23:02.726468 5112 raft_consensus.cc:2466] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ef8e002be104043af6666699b5396eb in term 1.
I20250905 02:23:02.728159 5152 leader_election.cc:304] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters:
W20250905 02:23:02.728964 5154 leader_election.cc:336] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
I20250905 02:23:02.729729 5242 leader_election.cc:290] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:02.730543 5404 raft_consensus.cc:2802] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 02:23:02.730911 5399 heartbeater.cc:344] Connected to a master server at 127.31.48.190:40685
I20250905 02:23:02.731045 5404 raft_consensus.cc:491] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 92b5c3bc59924a5f88755f068a0dea18)
I20250905 02:23:02.731361 5399 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:02.731611 5404 raft_consensus.cc:3058] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:02.732594 5399 heartbeater.cc:507] Master 127.31.48.190:40685 requested a full tablet report, sending...
I20250905 02:23:02.734724 4937 ts_manager.cc:194] Registered new tserver with Master: 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132:40795)
I20250905 02:23:02.736557 4937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48146
W20250905 02:23:02.738819 5403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 02:23:02.741703 5406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 02:23:02.742419 31938 server_base.cc:1047] running on GCE node
I20250905 02:23:02.743559 31938 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 02:23:02.743814 31938 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 02:23:02.744015 31938 hybrid_clock.cc:648] HybridClock initialized: now 1757038982743997 us; error 0 us; skew 500 ppm
I20250905 02:23:02.744731 31938 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 02:23:02.747972 31938 webserver.cc:480] Webserver started at http://127.31.48.129:42497/ using document root <none> and password file <none>
I20250905 02:23:02.748004 5404 raft_consensus.cc:513] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.748751 31938 fs_manager.cc:362] Metadata directory not provided
I20250905 02:23:02.748997 31938 fs_manager.cc:365] Using existing metadata directory in first data directory
I20250905 02:23:02.749392 5303 consensus_queue.cc:579] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Leader has been unable to successfully communicate with peer 92b5c3bc59924a5f88755f068a0dea18 for more than 1 seconds (1.511s)
I20250905 02:23:02.753795 5112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c0aed9a0934499ab22c6b93c4f9f915" candidate_uuid: "2ef8e002be104043af6666699b5396eb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5e5a0b8a2e3e47839db538a0abe12bee"
I20250905 02:23:02.754627 5112 raft_consensus.cc:3058] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:02.758322 31938 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.006s sys 0.003s
I20250905 02:23:02.751755 5404 leader_election.cc:290] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 election: Requested vote from peers 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789), 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
W20250905 02:23:02.759678 5154 leader_election.cc:336] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111)
I20250905 02:23:02.761617 5114 consensus_queue.cc:237] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:02.764350 5112 raft_consensus.cc:2466] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ef8e002be104043af6666699b5396eb in term 2.
I20250905 02:23:02.765724 5152 leader_election.cc:304] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ef8e002be104043af6666699b5396eb, 5e5a0b8a2e3e47839db538a0abe12bee; no voters: 92b5c3bc59924a5f88755f068a0dea18
I20250905 02:23:02.766846 5404 raft_consensus.cc:2802] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Leader election won for term 2
I20250905 02:23:02.768093 5404 raft_consensus.cc:695] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 2 LEADER]: Becoming Leader. State: Replica: 2ef8e002be104043af6666699b5396eb, State: Running, Role: LEADER
I20250905 02:23:02.769416 5411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 02:23:02.768958 5404 consensus_queue.cc:237] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.770332 31938 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.000s sys 0.006s
I20250905 02:23:02.770763 31938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root
uuid: "92b5c3bc59924a5f88755f068a0dea18"
format_stamp: "Formatted at 2025-09-05 02:22:58 on dist-test-slave-q32q"
I20250905 02:23:02.771144 31938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 02:23:02.774791 5186 raft_consensus.cc:1273] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:23:02.776899 5303 consensus_queue.cc:1035] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250905 02:23:02.796186 5247 raft_consensus.cc:2953] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:02.797681 4937 catalog_manager.cc:5582] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb reported cstate change: term changed from 1 to 2, leader changed from 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129) to 2ef8e002be104043af6666699b5396eb (127.31.48.131). New cstate: current_term: 2 leader_uuid: "2ef8e002be104043af6666699b5396eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: HEALTHY } } }
W20250905 02:23:02.801471 5080 consensus_peers.cc:489] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Couldn't send request to peer 92b5c3bc59924a5f88755f068a0dea18. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:02.805460 5186 raft_consensus.cc:2953] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:02.810889 4922 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4c99ff2310b74e33a06ecc309b900cf8 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250905 02:23:02.815086 4937 catalog_manager.cc:5582] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee reported cstate change: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 02:23:02.819644 5079 consensus_peers.cc:489] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132:40795): Couldn't send request to peer 50c6b99aa73c43dbae9f577b0c34ae5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c99ff2310b74e33a06ecc309b900cf8. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:02.853735 31938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 02:23:02.855059 31938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 02:23:02.856606 31938 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 02:23:02.859892 5423 ts_tablet_manager.cc:542] Loading tablet metadata (0/8 complete)
I20250905 02:23:02.872350 5114 consensus_queue.cc:237] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
W20250905 02:23:02.880067 5079 consensus_peers.cc:489] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132:40795): Couldn't send request to peer 50c6b99aa73c43dbae9f577b0c34ae5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27808973f5d24f91bfe2a2fd1495e472. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:02.882035 5186 raft_consensus.cc:1273] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 02:23:02.883769 5247 consensus_queue.cc:1035] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 02:23:02.884722 5080 consensus_peers.cc:489] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Couldn't send request to peer 92b5c3bc59924a5f88755f068a0dea18. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:02.891654 5303 raft_consensus.cc:2953] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:02.893134 5186 raft_consensus.cc:2953] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:02.902686 4922 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 27808973f5d24f91bfe2a2fd1495e472 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250905 02:23:02.904822 4936 catalog_manager.cc:5582] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb reported cstate change: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New cstate: current_term: 1 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:02.915952 31938 ts_tablet_manager.cc:579] Loaded tablet metadata (8 total tablets, 8 live tablets)
I20250905 02:23:02.916211 31938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.057s user 0.001s sys 0.001s
I20250905 02:23:02.916527 31938 ts_tablet_manager.cc:594] Registering tablets (0/8 complete)
I20250905 02:23:02.921417 5423 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:02.937459 5423 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:23:02.938448 5423 tablet_bootstrap.cc:492] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:02.939113 5423 ts_tablet_manager.cc:1397] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20250905 02:23:02.941735 5423 raft_consensus.cc:357] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.942605 5423 raft_consensus.cc:738] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:02.943286 5423 consensus_queue.cc:260] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.946270 5423 ts_tablet_manager.cc:1428] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.007s user 0.003s sys 0.005s
I20250905 02:23:02.947111 5423 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:02.960991 31938 ts_tablet_manager.cc:610] Registered 8 tablets
I20250905 02:23:02.961309 5423 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:02.961381 31938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.045s user 0.042s sys 0.002s
I20250905 02:23:02.962352 5423 tablet_bootstrap.cc:492] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:02.963325 5423 ts_tablet_manager.cc:1397] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250905 02:23:02.966107 5423 raft_consensus.cc:357] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.966840 5423 raft_consensus.cc:738] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:02.967595 5423 consensus_queue.cc:260] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.969977 5423 ts_tablet_manager.cc:1428] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250905 02:23:02.971042 5423 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:02.992496 5423 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:02.993701 5423 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:02.994554 5423 ts_tablet_manager.cc:1397] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.004s
I20250905 02:23:02.997931 5423 raft_consensus.cc:357] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:02.998807 5423 raft_consensus.cc:738] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:02.999769 5423 consensus_queue.cc:260] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.002555 5423 ts_tablet_manager.cc:1428] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20250905 02:23:03.003491 5423 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:03.014447 31938 rpc_server.cc:307] RPC server started. Bound to: 127.31.48.129:35789
I20250905 02:23:03.014604 5485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.31.48.129:35789 every 8 connection(s)
I20250905 02:23:03.020865 5423 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:03.022158 5423 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:03.023234 5423 ts_tablet_manager.cc:1397] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.012s
I20250905 02:23:03.025084 5423 raft_consensus.cc:357] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:03.025666 5423 raft_consensus.cc:738] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:03.026306 5423 consensus_queue.cc:260] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } }
I20250905 02:23:03.028025 5423 ts_tablet_manager.cc:1428] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 02:23:03.028817 5423 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:03.033215 5487 heartbeater.cc:344] Connected to a master server at 127.31.48.190:40685
I20250905 02:23:03.033597 5487 heartbeater.cc:461] Registering TS with master...
I20250905 02:23:03.034282 5487 heartbeater.cc:507] Master 127.31.48.190:40685 requested a full tablet report, sending...
I20250905 02:23:03.038801 4937 ts_manager.cc:194] Re-registered known tserver with Master: 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789)
I20250905 02:23:03.041714 5423 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:03.042693 5423 tablet_bootstrap.cc:492] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:03.043342 5423 ts_tablet_manager.cc:1397] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20250905 02:23:03.045776 5423 raft_consensus.cc:357] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.046482 5423 raft_consensus.cc:738] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:03.047108 5423 consensus_queue.cc:260] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.049189 5423 ts_tablet_manager.cc:1428] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250905 02:23:03.049510 4937 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:48150
I20250905 02:23:03.049986 5423 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:03.052531 5487 heartbeater.cc:499] Master 127.31.48.190:40685 was elected leader, sending a full tablet report...
I20250905 02:23:03.062948 5423 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:23:03.063953 5423 tablet_bootstrap.cc:492] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:03.064668 5423 ts_tablet_manager.cc:1397] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20250905 02:23:03.066923 5423 raft_consensus.cc:357] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.067700 5423 raft_consensus.cc:738] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:03.068295 5423 consensus_queue.cc:260] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.070020 5423 ts_tablet_manager.cc:1428] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 02:23:03.070778 5423 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:03.083108 5423 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:03.083771 5423 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:03.084239 5423 ts_tablet_manager.cc:1397] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20250905 02:23:03.085876 5423 raft_consensus.cc:357] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.086335 5423 raft_consensus.cc:738] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:03.086822 5423 consensus_queue.cc:260] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.088145 5423 ts_tablet_manager.cc:1428] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20250905 02:23:03.088943 5423 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap starting.
I20250905 02:23:03.099341 5423 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250905 02:23:03.100003 5423 tablet_bootstrap.cc:492] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Bootstrap complete.
I20250905 02:23:03.100524 5423 ts_tablet_manager.cc:1397] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20250905 02:23:03.102082 5423 raft_consensus.cc:357] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.102649 5423 raft_consensus.cc:738] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b5c3bc59924a5f88755f068a0dea18, State: Initialized, Role: FOLLOWER
I20250905 02:23:03.103106 5423 consensus_queue.cc:260] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.104494 5423 ts_tablet_manager.cc:1428] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250905 02:23:03.155417 5451 tablet_service.cc:1515] Processing DeleteTablet for tablet 539accf72c2a46dbbe2eae009b09d933 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41012
I20250905 02:23:03.157063 5495 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:03.157706 5495 raft_consensus.cc:2241] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:03.158084 5495 pending_rounds.cc:70] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Trying to abort 1 pending ops.
I20250905 02:23:03.158286 5495 pending_rounds.cc:77] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831666939293696 op_type: NO_OP noop_request { }
I20250905 02:23:03.158576 5495 raft_consensus.cc:2887] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:23:03.158864 5495 raft_consensus.cc:2270] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:03.160820 5495 ts_tablet_manager.cc:1905] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:03.164444 5186 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:23:03.167416 5303 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:23:03.174602 5495 ts_tablet_manager.cc:1918] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 02:23:03.175104 5495 log.cc:1199] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/wals/539accf72c2a46dbbe2eae009b09d933
I20250905 02:23:03.175869 5495 ts_tablet_manager.cc:1939] T 539accf72c2a46dbbe2eae009b09d933 P 92b5c3bc59924a5f88755f068a0dea18: Deleting consensus metadata
I20250905 02:23:03.180063 4924 catalog_manager.cc:4928] TS 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): tablet 539accf72c2a46dbbe2eae009b09d933 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:03.186657 5451 tablet_service.cc:1515] Processing DeleteTablet for tablet 1243692508e84769a132c2535f5d1c98 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41012
I20250905 02:23:03.187492 5495 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:03.188400 5495 raft_consensus.cc:2241] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:03.188840 5495 pending_rounds.cc:70] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Trying to abort 1 pending ops.
I20250905 02:23:03.189023 5495 pending_rounds.cc:77] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831666379538432 op_type: NO_OP noop_request { }
I20250905 02:23:03.189393 5495 raft_consensus.cc:2887] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:23:03.189728 5495 raft_consensus.cc:2270] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:03.190101 5461 raft_consensus.cc:3058] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:03.191778 5495 ts_tablet_manager.cc:1905] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:03.195811 5461 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:23:03.197203 5247 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:23:03.203899 5495 ts_tablet_manager.cc:1918] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 02:23:03.204425 5495 log.cc:1199] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/wals/1243692508e84769a132c2535f5d1c98
I20250905 02:23:03.205287 5495 ts_tablet_manager.cc:1939] T 1243692508e84769a132c2535f5d1c98 P 92b5c3bc59924a5f88755f068a0dea18: Deleting consensus metadata
I20250905 02:23:03.208112 4924 catalog_manager.cc:4928] TS 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): tablet 1243692508e84769a132c2535f5d1c98 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:03.215118 5451 tablet_service.cc:1515] Processing DeleteTablet for tablet a27c1e35910b4159a54954ff106d89cc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41012
I20250905 02:23:03.216018 5495 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:03.217033 5495 raft_consensus.cc:2241] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:03.217547 5495 pending_rounds.cc:70] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Trying to abort 1 pending ops.
I20250905 02:23:03.217793 5495 pending_rounds.cc:77] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7196831666967597056 op_type: NO_OP noop_request { }
I20250905 02:23:03.218150 5495 raft_consensus.cc:2887] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 02:23:03.218538 5495 raft_consensus.cc:2270] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:03.221717 5495 ts_tablet_manager.cc:1905] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:03.233786 5495 ts_tablet_manager.cc:1918] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 02:23:03.234400 5495 log.cc:1199] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/wals/a27c1e35910b4159a54954ff106d89cc
I20250905 02:23:03.234948 5495 ts_tablet_manager.cc:1939] T a27c1e35910b4159a54954ff106d89cc P 92b5c3bc59924a5f88755f068a0dea18: Deleting consensus metadata
I20250905 02:23:03.237110 4924 catalog_manager.cc:4928] TS 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): tablet a27c1e35910b4159a54954ff106d89cc (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:03.245111 5451 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a3bec2a01b4f58bb3270cd9e576291 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 02:23:02 UTC) from {username='slave'} at 127.0.0.1:41012
I20250905 02:23:03.245852 5495 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:03.246683 5495 raft_consensus.cc:2241] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:03.247174 5495 raft_consensus.cc:2270] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:03.251552 5495 ts_tablet_manager.cc:1905] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 02:23:03.252043 5461 raft_consensus.cc:3058] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Advancing to term 2
I20250905 02:23:03.258899 5461 raft_consensus.cc:1273] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Refusing update from remote peer 2ef8e002be104043af6666699b5396eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:23:03.260193 5404 consensus_queue.cc:1035] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:23:03.265815 5495 ts_tablet_manager.cc:1918] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 02:23:03.266252 5495 log.cc:1199] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-0-root/wals/76a3bec2a01b4f58bb3270cd9e576291
I20250905 02:23:03.268679 5495 ts_tablet_manager.cc:1939] T 76a3bec2a01b4f58bb3270cd9e576291 P 92b5c3bc59924a5f88755f068a0dea18: Deleting consensus metadata
I20250905 02:23:03.269713 5114 raft_consensus.cc:1273] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Refusing update from remote peer 2ef8e002be104043af6666699b5396eb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 02:23:03.271481 5502 consensus_queue.cc:1035] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 02:23:03.271543 4924 catalog_manager.cc:4928] TS 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): tablet 76a3bec2a01b4f58bb3270cd9e576291 (table dugtrio [id=f7e87748bb06486588b7a2a39a37a757]) successfully deleted
I20250905 02:23:03.276556 5503 ts_tablet_manager.cc:927] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Initiating tablet copy from peer 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:03.279075 5503 tablet_copy_client.cc:323] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.130:37139
I20250905 02:23:03.293453 5124 tablet_copy_service.cc:140] P 5e5a0b8a2e3e47839db538a0abe12bee: Received BeginTabletCopySession request for tablet 4c99ff2310b74e33a06ecc309b900cf8 from peer 50c6b99aa73c43dbae9f577b0c34ae5b ({username='slave'} at 127.0.0.1:41404)
I20250905 02:23:03.293969 5124 tablet_copy_service.cc:161] P 5e5a0b8a2e3e47839db538a0abe12bee: Beginning new tablet copy session on tablet 4c99ff2310b74e33a06ecc309b900cf8 from peer 50c6b99aa73c43dbae9f577b0c34ae5b at {username='slave'} at 127.0.0.1:41404: session id = 50c6b99aa73c43dbae9f577b0c34ae5b-4c99ff2310b74e33a06ecc309b900cf8
I20250905 02:23:03.299170 5124 tablet_copy_source_session.cc:215] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:23:03.301906 5503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c99ff2310b74e33a06ecc309b900cf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:03.312314 5503 tablet_copy_client.cc:806] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Starting download of 0 data blocks...
I20250905 02:23:03.312847 5503 tablet_copy_client.cc:670] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Starting download of 1 WAL segments...
I20250905 02:23:03.315922 5503 tablet_copy_client.cc:538] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:23:03.322116 5503 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap starting.
I20250905 02:23:03.335830 5503 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:03.336938 5503 tablet_bootstrap.cc:492] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap complete.
I20250905 02:23:03.337471 5503 ts_tablet_manager.cc:1397] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.005s
I20250905 02:23:03.339466 5503 raft_consensus.cc:357] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:03.340045 5503 raft_consensus.cc:738] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50c6b99aa73c43dbae9f577b0c34ae5b, State: Initialized, Role: LEARNER
I20250905 02:23:03.340595 5503 consensus_queue.cc:260] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:03.342293 5399 heartbeater.cc:499] Master 127.31.48.190:40685 was elected leader, sending a full tablet report...
I20250905 02:23:03.342630 5503 ts_tablet_manager.cc:1428] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250905 02:23:03.344276 5124 tablet_copy_service.cc:342] P 5e5a0b8a2e3e47839db538a0abe12bee: Request end of tablet copy session 50c6b99aa73c43dbae9f577b0c34ae5b-4c99ff2310b74e33a06ecc309b900cf8 received from {username='slave'} at 127.0.0.1:41404
I20250905 02:23:03.344645 5124 tablet_copy_service.cc:434] P 5e5a0b8a2e3e47839db538a0abe12bee: ending tablet copy session 50c6b99aa73c43dbae9f577b0c34ae5b-4c99ff2310b74e33a06ecc309b900cf8 on tablet 4c99ff2310b74e33a06ecc309b900cf8 with peer 50c6b99aa73c43dbae9f577b0c34ae5b
I20250905 02:23:03.352036 5461 raft_consensus.cc:2953] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:03.382258 5461 raft_consensus.cc:2953] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:03.453477 5503 ts_tablet_manager.cc:927] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Initiating tablet copy from peer 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:03.454754 5503 tablet_copy_client.cc:323] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.130:37139
I20250905 02:23:03.456063 5124 tablet_copy_service.cc:140] P 5e5a0b8a2e3e47839db538a0abe12bee: Received BeginTabletCopySession request for tablet 27808973f5d24f91bfe2a2fd1495e472 from peer 50c6b99aa73c43dbae9f577b0c34ae5b ({username='slave'} at 127.0.0.1:41404)
I20250905 02:23:03.456496 5124 tablet_copy_service.cc:161] P 5e5a0b8a2e3e47839db538a0abe12bee: Beginning new tablet copy session on tablet 27808973f5d24f91bfe2a2fd1495e472 from peer 50c6b99aa73c43dbae9f577b0c34ae5b at {username='slave'} at 127.0.0.1:41404: session id = 50c6b99aa73c43dbae9f577b0c34ae5b-27808973f5d24f91bfe2a2fd1495e472
I20250905 02:23:03.461441 5124 tablet_copy_source_session.cc:215] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:23:03.463976 5503 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27808973f5d24f91bfe2a2fd1495e472. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:03.474156 5503 tablet_copy_client.cc:806] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Starting download of 0 data blocks...
I20250905 02:23:03.474664 5503 tablet_copy_client.cc:670] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Starting download of 1 WAL segments...
I20250905 02:23:03.477506 5503 tablet_copy_client.cc:538] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:23:03.483425 5503 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap starting.
I20250905 02:23:03.497418 5503 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:03.498149 5503 tablet_bootstrap.cc:492] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap complete.
I20250905 02:23:03.499286 5503 ts_tablet_manager.cc:1397] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.004s
I20250905 02:23:03.501595 5503 raft_consensus.cc:357] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:03.502179 5503 raft_consensus.cc:738] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 50c6b99aa73c43dbae9f577b0c34ae5b, State: Initialized, Role: LEARNER
I20250905 02:23:03.502673 5503 consensus_queue.cc:260] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:03.504333 5503 ts_tablet_manager.cc:1428] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20250905 02:23:03.506356 5124 tablet_copy_service.cc:342] P 5e5a0b8a2e3e47839db538a0abe12bee: Request end of tablet copy session 50c6b99aa73c43dbae9f577b0c34ae5b-27808973f5d24f91bfe2a2fd1495e472 received from {username='slave'} at 127.0.0.1:41404
I20250905 02:23:03.506752 5124 tablet_copy_service.cc:434] P 5e5a0b8a2e3e47839db538a0abe12bee: ending tablet copy session 50c6b99aa73c43dbae9f577b0c34ae5b-27808973f5d24f91bfe2a2fd1495e472 on tablet 27808973f5d24f91bfe2a2fd1495e472 with peer 50c6b99aa73c43dbae9f577b0c34ae5b
I20250905 02:23:03.659649 5374 raft_consensus.cc:1215] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20250905 02:23:03.676694 5114 consensus_queue.cc:237] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:03.681950 5186 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20250905 02:23:03.682086 5079 consensus_peers.cc:489] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee -> Peer 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132:40795): Couldn't send request to peer 50c6b99aa73c43dbae9f577b0c34ae5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f302a9b041cb44408987319aac57e90b. This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:03.681950 5461 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250905 02:23:03.683431 5303 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:23:03.684336 5247 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:23:03.690387 5303 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:03.692018 5186 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:03.692902 5461 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
I20250905 02:23:03.703085 4937 catalog_manager.cc:5582] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb reported cstate change: config changed from index -1 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) added. New cstate: current_term: 2 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } } }
W20250905 02:23:03.707208 4985 auto_rebalancer.cc:254] failed to send replica move request: Invalid argument: Server with UUID 50c6b99aa73c43dbae9f577b0c34ae5b is already a member of the config. RaftConfig: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
W20250905 02:23:03.714478 4985 auto_rebalancer.cc:663] Could not move replica: Illegal state: 4c99ff2310b74e33a06ecc309b900cf8: source replica 92b5c3bc59924a5f88755f068a0dea18 does not have REPLACE attribute set
W20250905 02:23:03.714849 4985 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: 4c99ff2310b74e33a06ecc309b900cf8: source replica 92b5c3bc59924a5f88755f068a0dea18 does not have REPLACE attribute set
W20250905 02:23:03.722267 4985 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2c0aed9a0934499ab22c6b93c4f9f915, TS 5e5a0b8a2e3e47839db538a0abe12bee -> TS 50c6b99aa73c43dbae9f577b0c34ae5b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
W20250905 02:23:03.722519 4985 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2c0aed9a0934499ab22c6b93c4f9f915, TS 5e5a0b8a2e3e47839db538a0abe12bee -> TS 50c6b99aa73c43dbae9f577b0c34ae5b move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.793458 5114 consensus_queue.cc:237] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.797937 5461 raft_consensus.cc:1273] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:23:03.797987 5186 raft_consensus.cc:1273] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:23:03.799192 5256 consensus_queue.cc:1035] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:23:03.799926 5247 consensus_queue.cc:1035] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:23:03.805930 5256 raft_consensus.cc:2953] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } }
I20250905 02:23:03.807546 5186 raft_consensus.cc:2953] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } }
I20250905 02:23:03.813874 5461 raft_consensus.cc:2953] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } }
I20250905 02:23:03.815053 4922 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 4c99ff2310b74e33a06ecc309b900cf8 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:23:03.819826 4935 catalog_manager.cc:5582] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee reported cstate change: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New cstate: current_term: 1 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: HEALTHY } } }
I20250905 02:23:03.850235 5363 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c99ff2310b74e33a06ecc309b900cf8 with delete_type TABLET_DATA_TOMBSTONED (TS 50c6b99aa73c43dbae9f577b0c34ae5b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52572
I20250905 02:23:03.852284 5520 tablet_replica.cc:331] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: stopping tablet replica
I20250905 02:23:03.853209 5520 raft_consensus.cc:2241] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Raft consensus shutting down.
I20250905 02:23:03.853729 5520 raft_consensus.cc:2270] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Raft consensus is shut down!
I20250905 02:23:03.855798 5520 ts_tablet_manager.cc:1905] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:23:03.870286 5520 ts_tablet_manager.cc:1918] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250905 02:23:03.870620 5520 log.cc:1199] T 4c99ff2310b74e33a06ecc309b900cf8 P 50c6b99aa73c43dbae9f577b0c34ae5b: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root/wals/4c99ff2310b74e33a06ecc309b900cf8
I20250905 02:23:03.872133 4923 catalog_manager.cc:4928] TS 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132:40795): tablet 4c99ff2310b74e33a06ecc309b900cf8 (table test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]) successfully deleted
I20250905 02:23:03.957374 5114 consensus_queue.cc:237] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } }
I20250905 02:23:03.963053 5186 raft_consensus.cc:1273] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:23:03.963325 5461 raft_consensus.cc:1273] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250905 02:23:03.964483 5303 consensus_queue.cc:1035] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:23:03.965188 5256 consensus_queue.cc:1035] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 02:23:03.971776 5303 raft_consensus.cc:2953] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } }
I20250905 02:23:03.973656 5461 raft_consensus.cc:2953] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } }
I20250905 02:23:03.976478 5186 raft_consensus.cc:2953] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } } }
I20250905 02:23:03.980341 4922 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 27808973f5d24f91bfe2a2fd1495e472 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:23:03.983521 4935 catalog_manager.cc:5582] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee reported cstate change: config changed from index 2 to 3, NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) evicted. New cstate: current_term: 1 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } health_report { overall_health: HEALTHY } } }
I20250905 02:23:03.994966 5363 tablet_service.cc:1515] Processing DeleteTablet for tablet 27808973f5d24f91bfe2a2fd1495e472 with delete_type TABLET_DATA_TOMBSTONED (TS 50c6b99aa73c43dbae9f577b0c34ae5b not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:52572
I20250905 02:23:03.996845 5520 tablet_replica.cc:331] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: stopping tablet replica
I20250905 02:23:03.997648 5520 raft_consensus.cc:2241] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Raft consensus shutting down.
I20250905 02:23:03.998111 5520 raft_consensus.cc:2270] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b [term 1 LEARNER]: Raft consensus is shut down!
I20250905 02:23:04.000247 5520 ts_tablet_manager.cc:1905] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:23:04.012593 5520 ts_tablet_manager.cc:1918] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20250905 02:23:04.012980 5520 log.cc:1199] T 27808973f5d24f91bfe2a2fd1495e472 P 50c6b99aa73c43dbae9f577b0c34ae5b: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-3-root/wals/27808973f5d24f91bfe2a2fd1495e472
I20250905 02:23:04.014204 4923 catalog_manager.cc:4928] TS 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132:40795): tablet 27808973f5d24f91bfe2a2fd1495e472 (table test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]) successfully deleted
I20250905 02:23:04.256068 5523 ts_tablet_manager.cc:927] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: Initiating tablet copy from peer 5e5a0b8a2e3e47839db538a0abe12bee (127.31.48.130:37139)
I20250905 02:23:04.257735 5523 tablet_copy_client.cc:323] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Beginning tablet copy session from remote peer at address 127.31.48.130:37139
I20250905 02:23:04.259114 5124 tablet_copy_service.cc:140] P 5e5a0b8a2e3e47839db538a0abe12bee: Received BeginTabletCopySession request for tablet f302a9b041cb44408987319aac57e90b from peer 50c6b99aa73c43dbae9f577b0c34ae5b ({username='slave'} at 127.0.0.1:41404)
I20250905 02:23:04.259708 5124 tablet_copy_service.cc:161] P 5e5a0b8a2e3e47839db538a0abe12bee: Beginning new tablet copy session on tablet f302a9b041cb44408987319aac57e90b from peer 50c6b99aa73c43dbae9f577b0c34ae5b at {username='slave'} at 127.0.0.1:41404: session id = 50c6b99aa73c43dbae9f577b0c34ae5b-f302a9b041cb44408987319aac57e90b
I20250905 02:23:04.266104 5124 tablet_copy_source_session.cc:215] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 02:23:04.269332 5523 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f302a9b041cb44408987319aac57e90b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 02:23:04.280632 5523 tablet_copy_client.cc:806] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Starting download of 0 data blocks...
I20250905 02:23:04.281169 5523 tablet_copy_client.cc:670] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Starting download of 1 WAL segments...
I20250905 02:23:04.284505 5523 tablet_copy_client.cc:538] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 02:23:04.291599 5523 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap starting.
I20250905 02:23:04.308987 5523 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 02:23:04.309826 5523 tablet_bootstrap.cc:492] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: Bootstrap complete.
I20250905 02:23:04.310377 5523 ts_tablet_manager.cc:1397] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20250905 02:23:04.313870 5523 raft_consensus.cc:357] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:04.314472 5523 raft_consensus.cc:738] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 50c6b99aa73c43dbae9f577b0c34ae5b, State: Initialized, Role: LEARNER
I20250905 02:23:04.315024 5523 consensus_queue.cc:260] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: NON_VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: true } }
I20250905 02:23:04.317786 5523 ts_tablet_manager.cc:1428] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250905 02:23:04.319525 5124 tablet_copy_service.cc:342] P 5e5a0b8a2e3e47839db538a0abe12bee: Request end of tablet copy session 50c6b99aa73c43dbae9f577b0c34ae5b-f302a9b041cb44408987319aac57e90b received from {username='slave'} at 127.0.0.1:41404
I20250905 02:23:04.319936 5124 tablet_copy_service.cc:434] P 5e5a0b8a2e3e47839db538a0abe12bee: ending tablet copy session 50c6b99aa73c43dbae9f577b0c34ae5b-f302a9b041cb44408987319aac57e90b on tablet f302a9b041cb44408987319aac57e90b with peer 50c6b99aa73c43dbae9f577b0c34ae5b
I20250905 02:23:04.731770 5374 raft_consensus.cc:1215] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20250905 02:23:05.184826 5528 raft_consensus.cc:1062] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: attempting to promote NON_VOTER 50c6b99aa73c43dbae9f577b0c34ae5b to VOTER
I20250905 02:23:05.186853 5528 consensus_queue.cc:237] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } }
I20250905 02:23:05.193459 5461 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:23:05.193599 5186 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:23:05.193598 5374 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 LEARNER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250905 02:23:05.195122 5303 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:23:05.196106 5530 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:23:05.196849 5527 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250905 02:23:05.217674 5303 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.219357 5186 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.221058 5374 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.230862 4937 catalog_manager.cc:5582] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb reported cstate change: config changed from index 3 to 4, 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.236387 5461 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 50c6b99aa73c43dbae9f577b0c34ae5b (127.31.48.132) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "2ef8e002be104043af6666699b5396eb" member_type: VOTER last_known_addr { host: "127.31.48.131" port: 42239 } attrs { replace: true } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.251372 5114 consensus_queue.cc:237] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } }
I20250905 02:23:05.256018 5374 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 02:23:05.256867 5461 raft_consensus.cc:1273] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Refusing update from remote peer 5e5a0b8a2e3e47839db538a0abe12bee: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20250905 02:23:05.257596 5303 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:23:05.258481 5527 consensus_queue.cc:1035] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 02:23:05.265674 5530 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 2ef8e002be104043af6666699b5396eb (127.31.48.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.267145 5374 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 2ef8e002be104043af6666699b5396eb (127.31.48.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.269111 5461 raft_consensus.cc:2953] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 2ef8e002be104043af6666699b5396eb (127.31.48.131) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.278069 4922 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet f302a9b041cb44408987319aac57e90b with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 02:23:05.279218 4935 catalog_manager.cc:5582] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b reported cstate change: config changed from index 4 to 5, VOTER 2ef8e002be104043af6666699b5396eb (127.31.48.131) evicted. New cstate: current_term: 2 leader_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "92b5c3bc59924a5f88755f068a0dea18" member_type: VOTER last_known_addr { host: "127.31.48.129" port: 35789 } } peers { permanent_uuid: "5e5a0b8a2e3e47839db538a0abe12bee" member_type: VOTER last_known_addr { host: "127.31.48.130" port: 37139 } } peers { permanent_uuid: "50c6b99aa73c43dbae9f577b0c34ae5b" member_type: VOTER last_known_addr { host: "127.31.48.132" port: 40795 } attrs { promote: false } } }
I20250905 02:23:05.289106 5175 tablet_service.cc:1515] Processing DeleteTablet for tablet f302a9b041cb44408987319aac57e90b with delete_type TABLET_DATA_TOMBSTONED (TS 2ef8e002be104043af6666699b5396eb not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41564
I20250905 02:23:05.291035 5539 tablet_replica.cc:331] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:05.291805 5539 raft_consensus.cc:2241] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:05.292609 5539 raft_consensus.cc:2270] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:05.295581 5539 ts_tablet_manager.cc:1905] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 02:23:05.306396 5539 ts_tablet_manager.cc:1918] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20250905 02:23:05.306697 5539 log.cc:1199] T f302a9b041cb44408987319aac57e90b P 2ef8e002be104043af6666699b5396eb: Deleting WAL directory at /tmp/dist-test-taskBBwDIo/test-tmp/auto_rebalancer-test.0.AutoRebalancerTest.TestDeletedTables.1757038820745333-31938-0/minicluster-data/ts-2-root/wals/f302a9b041cb44408987319aac57e90b
I20250905 02:23:05.308017 4923 catalog_manager.cc:4928] TS 2ef8e002be104043af6666699b5396eb (127.31.48.131:42239): tablet f302a9b041cb44408987319aac57e90b (table test-workload [id=0a79151a0336426fae0e1c09cbfa7f80]) successfully deleted
W20250905 02:23:06.283982 4985 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 50c6b99aa73c43dbae9f577b0c34ae5b not available for placement
I20250905 02:23:07.052528 31938 tablet_server.cc:178] TabletServer@127.31.48.129:35789 shutting down...
I20250905 02:23:07.068313 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:23:07.069005 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:07.069609 31938 raft_consensus.cc:2241] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.070082 31938 raft_consensus.cc:2270] T 4c99ff2310b74e33a06ecc309b900cf8 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.071800 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:07.072311 31938 raft_consensus.cc:2241] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.072705 31938 raft_consensus.cc:2270] T 2c0aed9a0934499ab22c6b93c4f9f915 P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.074301 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:07.074769 31938 raft_consensus.cc:2241] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.075209 31938 raft_consensus.cc:2270] T 27808973f5d24f91bfe2a2fd1495e472 P 92b5c3bc59924a5f88755f068a0dea18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.076762 31938 tablet_replica.cc:331] stopping tablet replica
I20250905 02:23:07.077171 31938 raft_consensus.cc:2241] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.077541 31938 raft_consensus.cc:2270] T f302a9b041cb44408987319aac57e90b P 92b5c3bc59924a5f88755f068a0dea18 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.098549 31938 tablet_server.cc:195] TabletServer@127.31.48.129:35789 shutdown complete.
W20250905 02:23:07.103469 5154 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111) [suppressed 35 similar messages]
W20250905 02:23:07.106364 5154 consensus_peers.cc:489] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb -> Peer 92b5c3bc59924a5f88755f068a0dea18 (127.31.48.129:35789): Couldn't send request to peer 92b5c3bc59924a5f88755f068a0dea18. Status: Network error: Client connection negotiation failed: client connection to 127.31.48.129:35789: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 02:23:07.113343 31938 tablet_server.cc:178] TabletServer@127.31.48.130:0 shutting down...
I20250905 02:23:07.134341 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:23:07.134918 31938 tablet_replica.cc:331] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:07.135452 31938 raft_consensus.cc:2241] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.135906 31938 raft_consensus.cc:2270] T 2c0aed9a0934499ab22c6b93c4f9f915 P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.137828 31938 tablet_replica.cc:331] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:07.138336 31938 raft_consensus.cc:2241] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:07.139081 31938 raft_consensus.cc:2270] T 27808973f5d24f91bfe2a2fd1495e472 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.141042 31938 tablet_replica.cc:331] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:07.141686 31938 raft_consensus.cc:2241] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:07.142542 31938 raft_consensus.cc:2270] T 4c99ff2310b74e33a06ecc309b900cf8 P 5e5a0b8a2e3e47839db538a0abe12bee [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.144239 31938 tablet_replica.cc:331] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee: stopping tablet replica
I20250905 02:23:07.144743 31938 raft_consensus.cc:2241] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:23:07.145407 31938 raft_consensus.cc:2270] T f302a9b041cb44408987319aac57e90b P 5e5a0b8a2e3e47839db538a0abe12bee [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.167742 31938 tablet_server.cc:195] TabletServer@127.31.48.130:0 shutdown complete.
I20250905 02:23:07.185478 31938 tablet_server.cc:178] TabletServer@127.31.48.131:0 shutting down...
I20250905 02:23:07.208187 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:23:07.208801 31938 tablet_replica.cc:331] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:07.209349 31938 raft_consensus.cc:2241] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.209860 31938 raft_consensus.cc:2270] T 4c99ff2310b74e33a06ecc309b900cf8 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.211642 31938 tablet_replica.cc:331] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:07.212100 31938 raft_consensus.cc:2241] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 2 LEADER]: Raft consensus shutting down.
I20250905 02:23:07.212833 31938 raft_consensus.cc:2270] T 2c0aed9a0934499ab22c6b93c4f9f915 P 2ef8e002be104043af6666699b5396eb [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.214617 31938 tablet_replica.cc:331] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb: stopping tablet replica
I20250905 02:23:07.215039 31938 raft_consensus.cc:2241] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.215520 31938 raft_consensus.cc:2270] T 27808973f5d24f91bfe2a2fd1495e472 P 2ef8e002be104043af6666699b5396eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.238178 31938 tablet_server.cc:195] TabletServer@127.31.48.131:0 shutdown complete.
I20250905 02:23:07.255836 31938 tablet_server.cc:178] TabletServer@127.31.48.132:0 shutting down...
I20250905 02:23:07.274250 31938 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20250905 02:23:07.274847 31938 tablet_replica.cc:331] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b: stopping tablet replica
I20250905 02:23:07.275358 31938 raft_consensus.cc:2241] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 02:23:07.275907 31938 raft_consensus.cc:2270] T f302a9b041cb44408987319aac57e90b P 50c6b99aa73c43dbae9f577b0c34ae5b [term 2 FOLLOWER]: Raft consensus is shut down!
W20250905 02:23:07.286207 4985 auto_rebalancer.cc:227] Could not retrieve cluster info: Not found: tserver 2ef8e002be104043af6666699b5396eb not available for placement
I20250905 02:23:07.295521 31938 tablet_server.cc:195] TabletServer@127.31.48.132:0 shutdown complete.
I20250905 02:23:07.305917 31938 master.cc:561] Master@127.31.48.190:40685 shutting down...
I20250905 02:23:07.322420 31938 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 1 LEADER]: Raft consensus shutting down.
I20250905 02:23:07.322978 31938 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 02:23:07.323283 31938 tablet_replica.cc:331] T 00000000000000000000000000000000 P 83ab9b1d7ea9450f9d4fa09bd5afa1bc: stopping tablet replica
I20250905 02:23:07.341423 31938 master.cc:583] Master@127.31.48.190:40685 shutdown complete.
[ OK ] AutoRebalancerTest.TestDeletedTables (8871 ms)
[----------] 14 tests from AutoRebalancerTest (166473 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (166473 ms total)
[ PASSED ] 13 tests.
[ SKIPPED ] 1 test, listed below:
[ SKIPPED ] AutoRebalancerTest.TestMaxMovesPerServer
I20250905 02:23:07.382517 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:594: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250905 02:23:07.382792 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/ts_tablet_manager.cc:542: suppressed but not reported on 7 messages since previous log ~4 seconds ago
I20250905 02:23:07.382982 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:4671: suppressed but not reported on 19 messages since previous log ~4 seconds ago
I20250905 02:23:07.383193 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/master/catalog_manager.cc:5260: suppressed but not reported on 10 messages since previous log ~37 seconds ago
I20250905 02:23:07.383401 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 2 messages since previous log ~147 seconds ago
I20250905 02:23:07.383787 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tablet/tablet.cc:2378: suppressed but not reported on 37 messages since previous log ~15 seconds ago
I20250905 02:23:07.383956 31938 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 1 messages since previous log ~0 seconds ago
ThreadSanitizer: reported 1 warnings